Jul 29 20:24:32.841604 ip-10-0-136-123 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 20:24:33.277202 ip-10-0-136-123 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 20:24:33.277202 ip-10-0-136-123 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 20:24:33.277202 ip-10-0-136-123 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 20:24:33.277202 ip-10-0-136-123 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 20:24:33.277202 ip-10-0-136-123 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 20:24:33.277807 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.277208    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 20:24:33.282897 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282878    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:24:33.282897 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282893    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:24:33.282897 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282900    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:24:33.282897 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282904    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282907    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282910    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282912    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282916    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282919    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282922    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282925    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282928    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282938    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282941    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282946    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282949    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282951    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282954    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282957    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282960    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282962    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282965    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:24:33.283062 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282968    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282970    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282973    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282976    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282978    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282983    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282986    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282989    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282991    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282994    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282996    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.282999    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283002    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283005    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283008    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283012    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283014    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283020    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283022    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283025    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:24:33.283523 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283028    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283031    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283034    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283042    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283045    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283048    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283050    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283053    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283056    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283062    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283065    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283068    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283071    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283074    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283077    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283079    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283082    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283086    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283090    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:24:33.284016 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283093    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283097    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283102    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283105    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283108    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283111    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283113    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283116    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283119    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283122    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283125    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283134    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283136    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283139    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283142    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283147    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283150    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283159    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283163    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283166    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:24:33.284482 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283170    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283174    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283177    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283179    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283182    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283185    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283190    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283193    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283196    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283198    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283202    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283206    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283209    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283212    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283215    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283218    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283221    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:24:33.284959 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.283224    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285379    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285387    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285390    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285394    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285397    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285400    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285403    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285409    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285412    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285415    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285418    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:24:33.285418 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285421    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285429    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285432    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285435    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285438    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285441    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285443    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285446    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285449    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285454    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285457    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285459    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285462    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285465    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285468    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285471    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285473    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285477    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285479    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285482    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:24:33.285710 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285485    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285490    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285492    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285495    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285497    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285500    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285502    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285505    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285508    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285510    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285513    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285515    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285518    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285520    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285533    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285538    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285540    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285544    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285548    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:24:33.286232 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285551    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285554    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285556    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285559    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285561    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285564    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285570    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285572    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285577    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285580    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285584    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285587    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285590    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285592    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285595    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285598    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285601    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285604    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285607    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:24:33.286730 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285618    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285621    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285623    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285628    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285630    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285633    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285637    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285641    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285646    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285655    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285658    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285663    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285666    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285669    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285672    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285674    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285677    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285679    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285682    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285684    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:24:33.287370 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285687    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285690    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285692    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285695    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285700    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285703    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285705    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285708    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285711    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.285714    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286408    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286414    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286418    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286421    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286425    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286436    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286439    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286442    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286448    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:24:33.287854 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286451    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286456    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286466    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286469    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286472    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286475    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286478    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286480    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286483    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286486    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286488    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286493    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286496    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286499    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286501    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286505    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286508    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286511    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286513    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:24:33.288312 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286516    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286519    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286521    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286527    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286529    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286532    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286535    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286537    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286540    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286542    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286545    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286548    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286551    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286553    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286556    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286559    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286569    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286572    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286575    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286577    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:24:33.288793 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286580    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286582    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286585    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286588    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286590    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286593    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286596    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286598    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286601    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286606    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286608    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286611    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286613    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286616    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286620    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286622    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286625    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286627    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286630    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286633    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:24:33.289276 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286635    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286640    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286643    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286646    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286648    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286651    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286654    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286657    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286659    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286668    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286670    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286673    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286676    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286681    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286683    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286686    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286690    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286693    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286696    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:24:33.289851 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286699    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286702    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286704    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286707    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286709    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286712    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286714    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286719    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286722    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286725    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286727    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.286730    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286852    2585 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286859    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286868    2585 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286873    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286877    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286883    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286887    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286891    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286896    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 20:24:33.290300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286901    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286906    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286930    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286938    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.286942    2585 flags.go:64] FLAG: --cgroup-root=""
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287105    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287111    2585 flags.go:64] FLAG: --client-ca-file=""
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287115    2585 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287118    2585 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287126    2585 flags.go:64] FLAG: --cluster-domain=""
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287129    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287132    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287135    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287139    2585 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287144    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287147    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287151    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287154    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287157    2585 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287160    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287163    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287167    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287170    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287174    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 20:24:33.290809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287177    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287181    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287184    2585 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287187    2585 flags.go:64] FLAG: --enable-server="true"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287189    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287196    2585 flags.go:64] FLAG: --event-burst="100"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287199    2585 flags.go:64] FLAG: --event-qps="50"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287202    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287205    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287208    2585 flags.go:64] FLAG: --eviction-hard=""
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287212    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287215    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287218    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287221    2585 flags.go:64] FLAG: --eviction-soft=""
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287224    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287227    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287230    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287233    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287236    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287238    2585 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287241    2585 flags.go:64] FLAG: --feature-gates=""
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287245    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287248    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287251    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287254    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287260    2585 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 20:24:33.291414 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287265    2585 flags.go:64] FLAG: --help="false"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287268    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287273    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287276    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287279    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287283    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287286    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287289    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287293    2585 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287296    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287299    2585 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287304    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287308    2585 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287313    2585 flags.go:64] FLAG: --kube-reserved=""
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287316    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287320    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287323    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287339    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287342    2585 flags.go:64] FLAG: --lock-file=""
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287345    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287348    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287351    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287362    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 20:24:33.292089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287366    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287369    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287372    2585 flags.go:64] FLAG: --logging-format="text"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287375    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287378    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287381    2585 flags.go:64] FLAG: --manifest-url=""
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287384    2585 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287389    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287392    2585 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287396    2585 flags.go:64] FLAG: --max-pods="110"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287399    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287402    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287405    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287408    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287411    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287414    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287417    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287425    2585 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287428    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287431    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287434    2585 flags.go:64] FLAG: --pod-cidr=""
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287437    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287443    2585 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287445    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 20:24:33.292652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287448    2585 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287451    2585 flags.go:64] FLAG: --port="10250"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287454    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287457    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f153775c682d0178"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287461    2585 flags.go:64] FLAG: --qos-reserved=""
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287464    2585 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287467    2585 flags.go:64] FLAG: --register-node="true"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287470    2585 flags.go:64] FLAG: --register-with-taints=""
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287476    2585 flags.go:64] FLAG: --registry-burst="10"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287479    2585 flags.go:64] FLAG: --registry-qps="5"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287482    2585 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287485    2585 flags.go:64] FLAG: --reserved-memory=""
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287489    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287492    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287495    2585 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287498    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287504    2585 flags.go:64] FLAG: --runonce="false"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287507    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287510    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287513    2585 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287516    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287518    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287521    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287524    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287527    2585 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287530    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 20:24:33.293215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287533    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287536    2585 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287539    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287542    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287544    2585 flags.go:64] FLAG: --system-cgroups=""
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287547    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287553    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287556    2585 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287559    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287563    2585 flags.go:64] FLAG: --tls-min-version=""
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287566    2585 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287571    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287574    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287577    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287579    2585 flags.go:64] FLAG: --v="2"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287583    2585 flags.go:64] FLAG: --version="false"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287587    2585 flags.go:64] FLAG: --vmodule=""
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287591    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.287594    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287729    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287733    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287737    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287741    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287745    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:24:33.293848 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287748    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287752    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287756    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287759    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287762    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287765    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287768    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287770    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287773    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287776    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287778    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287781    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287784    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287787    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287789    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287792    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287795    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287797    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287800    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287804    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:24:33.294489 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287807    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287810    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287813    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287815    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287818    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287821    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287823    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287826    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287828    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287831    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287835    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287838    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287840    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287843    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287846    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287849    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287851    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287854    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287857    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287859    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:24:33.294973 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287862    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287865    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287868    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287870    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287873    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287875    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287878    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287881    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287883    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287886    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287889    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287892    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287895    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287898    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287900    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287903    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287905    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287907    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287911    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287913    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:24:33.295495 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287916    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287918    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287922    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287924    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287927    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287930    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287934    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287937    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287940    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287942    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287945    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287947    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287950    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287952    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287955    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287957    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287960    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287963    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287965    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:24:33.295977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287969    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287972    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287974    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287977    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287980    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287982    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287985    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287988    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287990    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287993    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287995    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.287998    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.288001    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.288003    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.288006    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:24:33.296452 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.288700    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.295433    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.295446    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.295467    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.295472    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295523    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295528    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295531    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295534    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295537    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295540    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295543    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295546    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295548    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295551    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295554    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295557    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295559    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295562    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295565    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:24:33.296823 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295567    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295570    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295572    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295575    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295578    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295581    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295583    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295586    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295588    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295591    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295593    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295596    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295598    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295600    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295604    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295607    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295610    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295613    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295615    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:24:33.297309 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295617    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295620    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295623    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295625    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295628    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295630    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295633    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295635    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295638    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295640    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295643    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295646    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295648    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295651    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295653    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295656    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295658    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295661    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295664    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295666    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:24:33.297780 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295669    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295671    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295674    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295678    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295682    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295685    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295688    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295692    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295695    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295698    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295700    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295703    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295705    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295708    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295711    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295714    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295716    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295718    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295721    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:24:33.298248 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295723    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295726    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295729    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295731    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295734    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295736    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295739    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295743    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295746    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295749    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295752    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295756    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295760    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295763    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295766    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295769    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295772    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295775    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295777    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:24:33.298705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295780    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295783    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295785    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295792    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295795    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295797    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295800    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.295805    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295908    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295912    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295915    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295918    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295920    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295923    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295925    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:24:33.299152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295928    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295931    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295933    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295935    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295938    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295940    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295943    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295946    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295949    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295951    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295953    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295956    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295958    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295961    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295963    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295965    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295968    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295971    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295973    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295976    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:24:33.299578 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295979    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295985    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295988    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295991    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295993    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295996    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.295998    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296001    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296004    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296006    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296009    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296011    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296014    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296016    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296019    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296021    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296024    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296026    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296028    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:24:33.300051 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296031    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296034    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296036    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296039    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296042    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296044    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296048    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296051    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296054    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296057    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296059    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296062    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296065    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296067    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296070    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296073    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296075    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296078    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296082    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:24:33.300536 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296085    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296088    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296091    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296093    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296096    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296099    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296101    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296104    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296106    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296109    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296111    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296114    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296117    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296119    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296122    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296124    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296127    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296129    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296132    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296135    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:24:33.301056 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296137    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296140    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296143    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296146    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296149    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296151    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296154    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296156    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296158    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296161    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296164    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296166    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296169    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:33.296171    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.296176    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 20:24:33.301565 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.296263    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 20:24:33.302018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.299035    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 20:24:33.302018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.300719    2585 server.go:1019] "Starting client certificate rotation"
Jul 29 20:24:33.302018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.300862    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 20:24:33.302018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.301711    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 20:24:33.303467 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.303453    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 20:24:33.304857 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.304836    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 20:24:33.317254 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.317237    2585 log.go:25] "Validated CRI v1 runtime API"
Jul 29 20:24:33.319954 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.319937    2585 log.go:25] "Validated CRI v1 image API"
Jul 29 20:24:33.320801 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.320776    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 20:24:33.323468 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.323450    2585 fs.go:135] Filesystem UUIDs: map[659f30ef-d678-42ea-b0f3-5801d977fb44:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b88866eb-8aed-4767-acca-f0ec8412755f:/dev/nvme0n1p4]
Jul 29 20:24:33.323530 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.323468    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 20:24:33.329583 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.329480    2585 manager.go:216] Machine: {Timestamp:2026-07-29 20:24:33.32803321 +0000 UTC m=+0.371725385 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100137 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ecbc09d70e3f58cf5cc2f619f0924 SystemUUID:ec2ecbc0-9d70-e3f5-8cf5-cc2f619f0924 BootID:07a9ecc3-f8db-4c8c-8159-0ce95197279a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5c:9e:3f:65:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5c:9e:3f:65:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:3b:48:44:ef:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 20:24:33.329583 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.329577    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 20:24:33.329714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.329667    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 20:24:33.333304 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.333287    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 20:24:33.333581 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.333555    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 20:24:33.333699 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.333586    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 20:24:33.334568 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.334558    2585 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 20:24:33.334605 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.334571    2585 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 20:24:33.334605 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.334578    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 20:24:33.335323 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.335312    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 20:24:33.336897 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.336888    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 20:24:33.340535 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.340525    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 20:24:33.340643 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.340634    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 20:24:33.340671 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.340661    2585 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 20:24:33.340705 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.340670    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 20:24:33.340705 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.340683    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 20:24:33.340705 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.340692    2585 kubelet.go:398] "Adding apiserver pod source"
Jul 29 20:24:33.340705 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.340699    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 20:24:33.341024 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.341014    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 20:24:33.341052 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.341031    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 20:24:33.343361 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.343348    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 20:24:33.345504 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.345490    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 20:24:33.345542 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.345522    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 20:24:33.347827 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347815    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 20:24:33.347891 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347831    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 20:24:33.347891 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347847    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 20:24:33.347891 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347853    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 20:24:33.347891 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347865    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 20:24:33.347891 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347873    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 20:24:33.347891 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347881    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 20:24:33.347891 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347886    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 20:24:33.347891 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347892    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 20:24:33.348153 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347898    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 20:24:33.348153 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347918    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 20:24:33.348153 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.347926    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 20:24:33.348727 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.348717    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 20:24:33.348727 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.348727    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 20:24:33.351555 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.351540    2585 server.go:1268] "Started kubelet"
Jul 29 20:24:33.351677 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.351642    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 20:24:33.351923 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.351764    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 20:24:33.352052 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.352031    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 20:24:33.352270 ip-10-0-136-123 systemd[1]: Started Kubernetes Kubelet.
Jul 29 20:24:33.354576 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.354000    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 20:24:33.354576 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.354025    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 20:24:33.354576 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.354065    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 20:24:33.354576 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.354075    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 20:24:33.354741 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.354718    2585 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 20:24:33.359898 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.359868    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 20:24:33.360645 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.359584    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-123.ec2.internal.18c6dc6f687d8181  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-123.ec2.internal,UID:ip-10-0-136-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-123.ec2.internal,},FirstTimestamp:2026-07-29 20:24:33.351516545 +0000 UTC m=+0.395208719,LastTimestamp:2026-07-29 20:24:33.351516545 +0000 UTC m=+0.395208719,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-123.ec2.internal,}"
Jul 29 20:24:33.362778 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.362761    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 20:24:33.362778 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.362770    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 20:24:33.362922 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.362901    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:33.363459 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363444    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 20:24:33.363459 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363444    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 20:24:33.363596 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363471    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 20:24:33.363596 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363583    2585 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 20:24:33.363596 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363590    2585 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 20:24:33.363751 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363745    2585 factory.go:153] Registering CRI-O factory
Jul 29 20:24:33.363801 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363756    2585 factory.go:223] Registration of the crio container factory successfully
Jul 29 20:24:33.363843 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363805    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 20:24:33.363843 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363816    2585 factory.go:55] Registering systemd factory
Jul 29 20:24:33.363843 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363825    2585 factory.go:223] Registration of the systemd container factory successfully
Jul 29 20:24:33.363974 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363847    2585 factory.go:103] Registering Raw factory
Jul 29 20:24:33.363974 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.363861    2585 manager.go:1196] Started watching for new ooms in manager
Jul 29 20:24:33.364310 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.364298    2585 manager.go:349] Starting recovery of all containers
Jul 29 20:24:33.365933 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.365910    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 20:24:33.366032 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.365985    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 20:24:33.373392 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.373376    2585 manager.go:354] Recovery completed
Jul 29 20:24:33.377302 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.377288    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:24:33.379110 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.379094    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:24:33.379188 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.379124    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:24:33.379188 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.379138    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:24:33.379649 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.379633    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 20:24:33.379649 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.379649    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 20:24:33.379781 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.379652    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:24:33.379781 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.379667    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 20:24:33.380502 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.380475    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:24:33.380555 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.380509    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:24:33.380555 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.380524    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:24:33.381932 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.381918    2585 policy_none.go:49] "None policy: Start"
Jul 29 20:24:33.381982 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.381940    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 20:24:33.381982 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.381956    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 20:24:33.383057 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.383045    2585 policy_none.go:47] "Start"
Jul 29 20:24:33.390928 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.390787    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-123.ec2.internal.18c6dc6f6a228902  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-123.ec2.internal,UID:ip-10-0-136-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-123.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-123.ec2.internal,},FirstTimestamp:2026-07-29 20:24:33.379109122 +0000 UTC m=+0.422801299,LastTimestamp:2026-07-29 20:24:33.379109122 +0000 UTC m=+0.422801299,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-123.ec2.internal,}"
Jul 29 20:24:33.401777 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.401756    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-578h9"
Jul 29 20:24:33.403855 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.403774    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-123.ec2.internal.18c6dc6f6a22e01e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-123.ec2.internal,UID:ip-10-0-136-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-123.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-123.ec2.internal,},FirstTimestamp:2026-07-29 20:24:33.379131422 +0000 UTC m=+0.422823602,LastTimestamp:2026-07-29 20:24:33.379131422 +0000 UTC m=+0.422823602,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-123.ec2.internal,}"
Jul 29 20:24:33.410085 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.410070    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-578h9"
Jul 29 20:24:33.412240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.412226    2585 manager.go:337] "Starting Device Plugin manager"
Jul 29 20:24:33.412306 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.412251    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 20:24:33.412306 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.412260    2585 server.go:87] "Starting device plugin registration server"
Jul 29 20:24:33.412470 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.412458    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 20:24:33.412524 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.412474    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 20:24:33.412583 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.412568    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 20:24:33.412920 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.412648    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 20:24:33.412920 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.412660    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 20:24:33.413025 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.412983    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 20:24:33.413025 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.413017    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:33.417050 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.416994    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-123.ec2.internal.18c6dc6f6a231213  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-123.ec2.internal,UID:ip-10-0-136-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-123.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-123.ec2.internal,},FirstTimestamp:2026-07-29 20:24:33.379144211 +0000 UTC m=+0.422836390,LastTimestamp:2026-07-29 20:24:33.379144211 +0000 UTC m=+0.422836390,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-123.ec2.internal,}"
Jul 29 20:24:33.513463 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.513444    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:24:33.514206 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.514178    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:24:33.514206 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.514208    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:24:33.514308 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.514222    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:24:33.514308 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.514251    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.516522 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.516500    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 20:24:33.517695 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.517680    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 20:24:33.517834 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.517718    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 20:24:33.517834 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.517742    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 20:24:33.517834 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.517776    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 20:24:33.520579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.520564    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:24:33.540221 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.540175    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.540221 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.540200    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-123.ec2.internal\": node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:33.617835 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.617803    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"]
Jul 29 20:24:33.617944 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.617878    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:24:33.618596 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.618578    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:24:33.618656 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.618607    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:24:33.618656 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.618617    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:24:33.619797 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.619784    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:24:33.619929 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.619915    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.619989 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.619948    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:24:33.620363 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.620349    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:24:33.620435 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.620372    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:24:33.620435 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.620381    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:24:33.620435 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.620395    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:24:33.620435 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.620418    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:24:33.620564 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.620428    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:24:33.621529 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.621512    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.621594 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.621545    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:24:33.622196 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.622181    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:24:33.622267 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.622202    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:24:33.622267 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.622213    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:24:33.634565 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.634550    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:33.659952 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.659935    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-123.ec2.internal\" not found" node="ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.664116 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.664101    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-123.ec2.internal\" not found" node="ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.665506 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.665490    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d6135b471ad91bdc496cb5531f8ee37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"0d6135b471ad91bdc496cb5531f8ee37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.665580 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.665519    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d6135b471ad91bdc496cb5531f8ee37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"0d6135b471ad91bdc496cb5531f8ee37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.665580 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.665545    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f03ef72ad08cd4666542db4e9e3bb71-config\") pod \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" (UID: \"1f03ef72ad08cd4666542db4e9e3bb71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.735368 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.735351    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:33.766132 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.766113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d6135b471ad91bdc496cb5531f8ee37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"0d6135b471ad91bdc496cb5531f8ee37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.766219 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.766135    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d6135b471ad91bdc496cb5531f8ee37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"0d6135b471ad91bdc496cb5531f8ee37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.766219 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.766151    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f03ef72ad08cd4666542db4e9e3bb71-config\") pod \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" (UID: \"1f03ef72ad08cd4666542db4e9e3bb71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.766219 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.766189    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f03ef72ad08cd4666542db4e9e3bb71-config\") pod \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" (UID: \"1f03ef72ad08cd4666542db4e9e3bb71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.766219 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.766211    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d6135b471ad91bdc496cb5531f8ee37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"0d6135b471ad91bdc496cb5531f8ee37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.766447 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.766245    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d6135b471ad91bdc496cb5531f8ee37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"0d6135b471ad91bdc496cb5531f8ee37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.835452 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.835401    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:33.936006 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:33.935983    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:33.962170 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.962142    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:33.968037 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:33.968013    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:34.036846 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:34.036820    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:34.137344 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:34.137286    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:34.237823 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:34.237803    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:34.300293 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.300267    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 20:24:34.300786 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.300444    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 20:24:34.338555 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:34.338533    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:34.415982 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.415906    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 20:19:34 +0000 UTC" deadline="2028-03-07 13:35:31.686619119 +0000 UTC"
Jul 29 20:24:34.415982 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.415935    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14081h10m57.270687445s"
Jul 29 20:24:34.427197 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.427170    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:24:34.428511 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.428494    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:24:34.428601 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.428523    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:24:34.428601 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.428538    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:24:34.438818 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:34.438795    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:34.447050 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:34.447023    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6135b471ad91bdc496cb5531f8ee37.slice/crio-8e259dc8a540c6468703330be6a0426426b03c9250b8828945e5c1790a2b357a WatchSource:0}: Error finding container 8e259dc8a540c6468703330be6a0426426b03c9250b8828945e5c1790a2b357a: Status 404 returned error can't find the container with id 8e259dc8a540c6468703330be6a0426426b03c9250b8828945e5c1790a2b357a
Jul 29 20:24:34.447870 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:34.447844    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f03ef72ad08cd4666542db4e9e3bb71.slice/crio-675d75e569bac5966e93386741501b4e780145ab9272a9d15c5591ec0bf82331 WatchSource:0}: Error finding container 675d75e569bac5966e93386741501b4e780145ab9272a9d15c5591ec0bf82331: Status 404 returned error can't find the container with id 675d75e569bac5966e93386741501b4e780145ab9272a9d15c5591ec0bf82331
Jul 29 20:24:34.451709 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.451694    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:24:34.523280 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.523238    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" event={"ID":"0d6135b471ad91bdc496cb5531f8ee37","Type":"ContainerStarted","Data":"8e259dc8a540c6468703330be6a0426426b03c9250b8828945e5c1790a2b357a"}
Jul 29 20:24:34.524318 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.524298    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" event={"ID":"1f03ef72ad08cd4666542db4e9e3bb71","Type":"ContainerStarted","Data":"675d75e569bac5966e93386741501b4e780145ab9272a9d15c5591ec0bf82331"}
Jul 29 20:24:34.539529 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:34.539511    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found"
Jul 29 20:24:34.605335 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.605320    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:24:34.663918 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.663896    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:34.674189 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.674149    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 20:24:34.676046 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.676034    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal"
Jul 29 20:24:34.691734 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.691716    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 20:24:34.866197 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.866174    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:24:34.898982 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:34.898960    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:24:35.342273 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.342251    2585 apiserver.go:52] "Watching apiserver"
Jul 29 20:24:35.350351 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.350307    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 20:24:35.350724 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.350700    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-29rwk","openshift-dns/node-resolver-qm8fd","openshift-image-registry/node-ca-k58wb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal","openshift-multus/network-metrics-daemon-7sqwj","openshift-network-diagnostics/network-check-target-zmnd7","kube-system/global-pull-secret-syncer-9xh8g","kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal","openshift-multus/multus-6hgg4","openshift-multus/multus-additional-cni-plugins-7z62g","openshift-network-operator/iptables-alerter-bsvn9","openshift-ovn-kubernetes/ovnkube-node-5lprd","kube-system/konnectivity-agent-h9t4t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"]
Jul 29 20:24:35.354090 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.354066    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.355619 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.355367    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:35.357494 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.357471    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 20:24:35.357601 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.357562    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 20:24:35.357671 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.357648    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 20:24:35.357739 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.357644    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.357830 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.357808    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 20:24:35.358175 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.358139    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 20:24:35.358323 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.358297    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ldwqz\""
Jul 29 20:24:35.358419 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.358324    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 20:24:35.359048 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.359024    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 20:24:35.359152 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.359059    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b5l2g\""
Jul 29 20:24:35.359285 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.359220    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.360416 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.360395    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 20:24:35.360597 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.360578    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s4w5v\""
Jul 29 20:24:35.360597 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.360649    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 20:24:35.360597 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.360739    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 20:24:35.360949 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.360790    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:35.360949 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.360854    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:35.362182 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.362149    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 20:24:35.362267 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.362246    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:35.362383 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.362318    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:35.363340 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.363251    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 20:24:35.363745 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.363723    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 20:24:35.364766 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.363730    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-shf7z\""
Jul 29 20:24:35.364766 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.364053    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.364766 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.364326    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 20:24:35.366026 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.365568    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:35.366026 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.365668    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:35.369715 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.368981    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.369715 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.369634    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 20:24:35.369888 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.369715    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lfp9q\""
Jul 29 20:24:35.371983 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.371962    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6slhk\""
Jul 29 20:24:35.372143 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.372125    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 20:24:35.372216 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.372200    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 20:24:35.372553 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.372530    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.374001 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.373981    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.375168 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375147    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-var-lib-cni-bin\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.375271 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375181    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-run-multus-certs\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.375271 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375209    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-etc-kubernetes\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.375271 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375246    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375277    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-registration-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375302    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-device-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375326    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-etc-selinux\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375366    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54nr9\" (UniqueName: \"kubernetes.io/projected/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-kube-api-access-54nr9\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375390    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-system-cni-dir\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375416    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-multus-cni-dir\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375441    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-multus-conf-dir\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375460    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375480    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f01f25d-0021-40f3-a536-01a921d1557d-multus-daemon-config\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.375518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375508    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-cnibin\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375558    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a78b2c7b-217a-4490-921d-84357434b580-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375611    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4b6\" (UniqueName: \"kubernetes.io/projected/1f01f25d-0021-40f3-a536-01a921d1557d-kube-api-access-tx4b6\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375648    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7cf47463-470a-4c86-8301-a838734a4963-serviceca\") pod \"node-ca-k58wb\" (UID: \"7cf47463-470a-4c86-8301-a838734a4963\") " pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375674    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375724    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f01f25d-0021-40f3-a536-01a921d1557d-cni-binary-copy\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375749    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375775    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jx6s\" (UniqueName: \"kubernetes.io/projected/a78b2c7b-217a-4490-921d-84357434b580-kube-api-access-4jx6s\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cf47463-470a-4c86-8301-a838734a4963-host\") pod \"node-ca-k58wb\" (UID: \"7cf47463-470a-4c86-8301-a838734a4963\") " pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375893    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375921    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-os-release\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.376012 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.375948    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-multus-socket-dir-parent\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376015    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a78b2c7b-217a-4490-921d-84357434b580-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376092    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376095    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-socket-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376146    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xr5b6\""
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376233    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376145    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-cnibin\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376271    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-system-cni-dir\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376298    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a78b2c7b-217a-4490-921d-84357434b580-cni-binary-copy\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376324    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7hg\" (UniqueName: \"kubernetes.io/projected/94bf5594-165a-4efb-ba8c-a628ea34eabf-kube-api-access-8r7hg\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376384    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-run-k8s-cni-cncf-io\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376411    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-var-lib-cni-multus\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376437    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-var-lib-kubelet\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376460    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-os-release\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376485    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26eecc01-788c-477e-b50a-5553413d3dd8-agent-certs\") pod \"konnectivity-agent-h9t4t\" (UID: \"26eecc01-788c-477e-b50a-5553413d3dd8\") " pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:35.376508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376505    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26eecc01-788c-477e-b50a-5553413d3dd8-konnectivity-ca\") pod \"konnectivity-agent-h9t4t\" (UID: \"26eecc01-788c-477e-b50a-5553413d3dd8\") " pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:35.377154 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376529    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66qpt\" (UniqueName: \"kubernetes.io/projected/7cf47463-470a-4c86-8301-a838734a4963-kube-api-access-66qpt\") pod \"node-ca-k58wb\" (UID: \"7cf47463-470a-4c86-8301-a838734a4963\") " pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.377154 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376552    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-run-netns\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.377154 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376613    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-hostroot\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.377154 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.376663    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-sys-fs\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.377515 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.377485    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 20:24:35.377515 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.377493    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:24:35.377515 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.377511    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 20:24:35.377697 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.377677    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mxl9m\""
Jul 29 20:24:35.377929 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.377914    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 20:24:35.378011 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.377984    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 20:24:35.378663 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.378644    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 20:24:35.378663 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.378655    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 20:24:35.378809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.378776    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8sxln\""
Jul 29 20:24:35.380154 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.380133    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 20:24:35.380265 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.380211    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 20:24:35.380265 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.380221    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 20:24:35.401576 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.401554    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n6wqn"
Jul 29 20:24:35.410279 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.410251    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n6wqn"
Jul 29 20:24:35.465134 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.465091    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 20:24:35.476836 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.476798    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-run-k8s-cni-cncf-io\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.476972 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.476841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-var-lib-cni-multus\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.476972 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.476873    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-os-release\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.476972 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.476900    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66qpt\" (UniqueName: \"kubernetes.io/projected/7cf47463-470a-4c86-8301-a838734a4963-kube-api-access-66qpt\") pod \"node-ca-k58wb\" (UID: \"7cf47463-470a-4c86-8301-a838734a4963\") " pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.476972 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.476927    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-hostroot\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.476972 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.476939    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-run-k8s-cni-cncf-io\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.476972 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.476953    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-sys-fs\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.476985    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477008    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-hostroot\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477020    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps6cf\" (UniqueName: \"kubernetes.io/projected/c6c5f10b-41ce-40fc-b5dd-d49bdcca7763-kube-api-access-ps6cf\") pod \"iptables-alerter-bsvn9\" (UID: \"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763\") " pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.476955    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-var-lib-cni-multus\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477054    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-lib-modules\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477080    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-run-systemd\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477098    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-sys-fs\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477110    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-run-openvswitch\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477136    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-var-lib-cni-bin\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477166    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-os-release\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477169    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-run-multus-certs\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477211    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-etc-selinux\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477215    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-run-multus-certs\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477238    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a78b2c7b-217a-4490-921d-84357434b580-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477314    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6c5f10b-41ce-40fc-b5dd-d49bdcca7763-iptables-alerter-script\") pod \"iptables-alerter-bsvn9\" (UID: \"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763\") " pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.477355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477345    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4k2\" (UniqueName: \"kubernetes.io/projected/9191e46f-8231-4d1c-8104-c679ad0e2eeb-kube-api-access-mk4k2\") pod \"node-resolver-qm8fd\" (UID: \"9191e46f-8231-4d1c-8104-c679ad0e2eeb\") " pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477353    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-etc-selinux\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477415    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-run\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477444    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-kubelet\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477498    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-var-lib-cni-bin\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477551    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33d12f3d-f13d-4f39-9fd3-c77414affabc-env-overrides\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477582    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7cf47463-470a-4c86-8301-a838734a4963-serviceca\") pod \"node-ca-k58wb\" (UID: \"7cf47463-470a-4c86-8301-a838734a4963\") " pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477653    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9191e46f-8231-4d1c-8104-c679ad0e2eeb-tmp-dir\") pod \"node-resolver-qm8fd\" (UID: \"9191e46f-8231-4d1c-8104-c679ad0e2eeb\") " pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477725    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-run-ovn\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477799    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477863    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jx6s\" (UniqueName: \"kubernetes.io/projected/a78b2c7b-217a-4490-921d-84357434b580-kube-api-access-4jx6s\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477903    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a78b2c7b-217a-4490-921d-84357434b580-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477920    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgtb\" (UniqueName: \"kubernetes.io/projected/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-kube-api-access-tjgtb\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.477972    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-systemd-units\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478248 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478065    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-log-socket\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478264    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7cf47463-470a-4c86-8301-a838734a4963-serviceca\") pod \"node-ca-k58wb\" (UID: \"7cf47463-470a-4c86-8301-a838734a4963\") " pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478278    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-cni-netd\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478311    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478341    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-os-release\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478379    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-multus-socket-dir-parent\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478426    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a78b2c7b-217a-4490-921d-84357434b580-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478449    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-run-netns\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478499    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33d12f3d-f13d-4f39-9fd3-c77414affabc-ovnkube-script-lib\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478526    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-system-cni-dir\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478549    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-multus-socket-dir-parent\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478549    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-node-log\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478591    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478629    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-var-lib-kubelet\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478668    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26eecc01-788c-477e-b50a-5553413d3dd8-agent-certs\") pod \"konnectivity-agent-h9t4t\" (UID: \"26eecc01-788c-477e-b50a-5553413d3dd8\") " pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478706    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-os-release\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.478730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478703    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26eecc01-788c-477e-b50a-5553413d3dd8-konnectivity-ca\") pod \"konnectivity-agent-h9t4t\" (UID: \"26eecc01-788c-477e-b50a-5553413d3dd8\") " pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478753    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-system-cni-dir\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478819    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-tuned\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478850    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-slash\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478900    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-var-lib-kubelet\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.478971    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqdkb\" (UniqueName: \"kubernetes.io/projected/33d12f3d-f13d-4f39-9fd3-c77414affabc-kube-api-access-wqdkb\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479019    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-run-netns\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479021    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a78b2c7b-217a-4490-921d-84357434b580-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479059    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-sysctl-d\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479111    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-host-run-netns\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-sys\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.479230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479195    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-cni-bin\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-etc-kubernetes\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479243    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479316    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-etc-kubernetes\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479263    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.479376    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479408    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-registration-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479414    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26eecc01-788c-477e-b50a-5553413d3dd8-konnectivity-ca\") pod \"konnectivity-agent-h9t4t\" (UID: \"26eecc01-788c-477e-b50a-5553413d3dd8\") " pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.479500    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs podName:94bf5594-165a-4efb-ba8c-a628ea34eabf nodeName:}" failed. No retries permitted until 2026-07-29 20:24:35.97945917 +0000 UTC m=+3.023151358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs") pod "network-metrics-daemon-7sqwj" (UID: "94bf5594-165a-4efb-ba8c-a628ea34eabf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479508    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-registration-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.479579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479525    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-device-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.479997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479562    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54nr9\" (UniqueName: \"kubernetes.io/projected/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-kube-api-access-54nr9\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.479997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479619    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6c5f10b-41ce-40fc-b5dd-d49bdcca7763-host-slash\") pod \"iptables-alerter-bsvn9\" (UID: \"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763\") " pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.479997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479643    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-tmp\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.479997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479672    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-system-cni-dir\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.479997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479587    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-device-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.479997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.479697    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-multus-cni-dir\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.480249 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.480219    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-system-cni-dir\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.480297 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.480260    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-multus-conf-dir\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.480297 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.480290    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f01f25d-0021-40f3-a536-01a921d1557d-multus-daemon-config\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.480377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.480335    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-cnibin\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.480377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.480362    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-sysconfig\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.480492 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.480389    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-kubernetes\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.480947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.480920    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f01f25d-0021-40f3-a536-01a921d1557d-multus-daemon-config\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.480947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.480919    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-cnibin\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.481117 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481000    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4b6\" (UniqueName: \"kubernetes.io/projected/1f01f25d-0021-40f3-a536-01a921d1557d-kube-api-access-tx4b6\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.481117 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481046    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0c1248c-5bb1-4cd3-bade-adc29339979c-kubelet-config\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:35.481117 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481059    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-multus-cni-dir\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.481117 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481080    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-host\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.481360 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481148    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-multus-conf-dir\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.481360 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481188    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-etc-openvswitch\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.481360 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481223    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f01f25d-0021-40f3-a536-01a921d1557d-cni-binary-copy\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.481360 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481249    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.481360 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481273    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cf47463-470a-4c86-8301-a838734a4963-host\") pod \"node-ca-k58wb\" (UID: \"7cf47463-470a-4c86-8301-a838734a4963\") " pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.481360 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481302    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.481360 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481332    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-modprobe-d\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481377    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-sysctl-conf\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-systemd\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481428    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33d12f3d-f13d-4f39-9fd3-c77414affabc-ovn-node-metrics-cert\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481453    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-socket-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481464    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a78b2c7b-217a-4490-921d-84357434b580-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481474    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0c1248c-5bb1-4cd3-bade-adc29339979c-dbus\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481515    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-var-lib-openvswitch\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481540    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cf47463-470a-4c86-8301-a838734a4963-host\") pod \"node-ca-k58wb\" (UID: \"7cf47463-470a-4c86-8301-a838734a4963\") " pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481543    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33d12f3d-f13d-4f39-9fd3-c77414affabc-ovnkube-config\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481578    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-cnibin\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481630    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a78b2c7b-217a-4490-921d-84357434b580-cni-binary-copy\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481639    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481665    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7hg\" (UniqueName: \"kubernetes.io/projected/94bf5594-165a-4efb-ba8c-a628ea34eabf-kube-api-access-8r7hg\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481698    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f01f25d-0021-40f3-a536-01a921d1557d-cni-binary-copy\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481709    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9191e46f-8231-4d1c-8104-c679ad0e2eeb-hosts-file\") pod \"node-resolver-qm8fd\" (UID: \"9191e46f-8231-4d1c-8104-c679ad0e2eeb\") " pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481737    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-var-lib-kubelet\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.481746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481745    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-socket-dir\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.482533 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.481781    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f01f25d-0021-40f3-a536-01a921d1557d-cnibin\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.482533 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.482073    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a78b2c7b-217a-4490-921d-84357434b580-cni-binary-copy\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.484149 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.484124    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26eecc01-788c-477e-b50a-5553413d3dd8-agent-certs\") pod \"konnectivity-agent-h9t4t\" (UID: \"26eecc01-788c-477e-b50a-5553413d3dd8\") " pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:35.487582 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.487561    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:24:35.487887 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.487586    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:24:35.487887 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.487600    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ngk9r for pod openshift-network-diagnostics/network-check-target-zmnd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:35.487887 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.487666    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r podName:a0d385ca-876f-4748-a57c-8e0c3f5ca7cc nodeName:}" failed. No retries permitted until 2026-07-29 20:24:35.987647304 +0000 UTC m=+3.031339481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ngk9r" (UniqueName: "kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r") pod "network-check-target-zmnd7" (UID: "a0d385ca-876f-4748-a57c-8e0c3f5ca7cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:35.489915 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.489817    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jx6s\" (UniqueName: \"kubernetes.io/projected/a78b2c7b-217a-4490-921d-84357434b580-kube-api-access-4jx6s\") pod \"multus-additional-cni-plugins-7z62g\" (UID: \"a78b2c7b-217a-4490-921d-84357434b580\") " pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.496125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.493369    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54nr9\" (UniqueName: \"kubernetes.io/projected/e5a2a86a-79da-4c24-b0e1-0a3a3b510b73-kube-api-access-54nr9\") pod \"aws-ebs-csi-driver-node-z2r4c\" (UID: \"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.496125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.495351    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4b6\" (UniqueName: \"kubernetes.io/projected/1f01f25d-0021-40f3-a536-01a921d1557d-kube-api-access-tx4b6\") pod \"multus-6hgg4\" (UID: \"1f01f25d-0021-40f3-a536-01a921d1557d\") " pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.496125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.496001    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66qpt\" (UniqueName: \"kubernetes.io/projected/7cf47463-470a-4c86-8301-a838734a4963-kube-api-access-66qpt\") pod \"node-ca-k58wb\" (UID: \"7cf47463-470a-4c86-8301-a838734a4963\") " pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.502766 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.502715    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7hg\" (UniqueName: \"kubernetes.io/projected/94bf5594-165a-4efb-ba8c-a628ea34eabf-kube-api-access-8r7hg\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:35.582701 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582666    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6c5f10b-41ce-40fc-b5dd-d49bdcca7763-host-slash\") pod \"iptables-alerter-bsvn9\" (UID: \"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763\") " pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.582888 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582713    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-tmp\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.582888 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582750    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-sysconfig\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.582888 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582778    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-kubernetes\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.582888 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582803    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0c1248c-5bb1-4cd3-bade-adc29339979c-kubelet-config\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:35.582888 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582824    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-host\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.582888 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582847    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-etc-openvswitch\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.582888 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-modprobe-d\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582897    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-sysctl-conf\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582919    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-systemd\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582941    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33d12f3d-f13d-4f39-9fd3-c77414affabc-ovn-node-metrics-cert\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582966    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0c1248c-5bb1-4cd3-bade-adc29339979c-dbus\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582991    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-var-lib-openvswitch\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.582996    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6c5f10b-41ce-40fc-b5dd-d49bdcca7763-host-slash\") pod \"iptables-alerter-bsvn9\" (UID: \"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763\") " pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583015    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33d12f3d-f13d-4f39-9fd3-c77414affabc-ovnkube-config\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583045    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9191e46f-8231-4d1c-8104-c679ad0e2eeb-hosts-file\") pod \"node-resolver-qm8fd\" (UID: \"9191e46f-8231-4d1c-8104-c679ad0e2eeb\") " pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583068    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-var-lib-kubelet\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583100    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583127    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps6cf\" (UniqueName: \"kubernetes.io/projected/c6c5f10b-41ce-40fc-b5dd-d49bdcca7763-kube-api-access-ps6cf\") pod \"iptables-alerter-bsvn9\" (UID: \"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763\") " pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-lib-modules\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583175    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-run-systemd\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583202    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-run-openvswitch\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6c5f10b-41ce-40fc-b5dd-d49bdcca7763-iptables-alerter-script\") pod \"iptables-alerter-bsvn9\" (UID: \"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763\") " pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4k2\" (UniqueName: \"kubernetes.io/projected/9191e46f-8231-4d1c-8104-c679ad0e2eeb-kube-api-access-mk4k2\") pod \"node-resolver-qm8fd\" (UID: \"9191e46f-8231-4d1c-8104-c679ad0e2eeb\") " pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.583283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583258    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-sysctl-conf\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583280    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-run\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583303    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-kubelet\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583309    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-sysconfig\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583328    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33d12f3d-f13d-4f39-9fd3-c77414affabc-env-overrides\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583369    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-kubernetes\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583362    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9191e46f-8231-4d1c-8104-c679ad0e2eeb-tmp-dir\") pod \"node-resolver-qm8fd\" (UID: \"9191e46f-8231-4d1c-8104-c679ad0e2eeb\") " pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583434    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-run-ovn\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583441    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0c1248c-5bb1-4cd3-bade-adc29339979c-kubelet-config\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583500    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-host\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583510    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-run-ovn\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgtb\" (UniqueName: \"kubernetes.io/projected/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-kube-api-access-tjgtb\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583565    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-systemd-units\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583575    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-systemd\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583592    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-log-socket\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583621    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-cni-netd\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583650    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583683    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-run-netns\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.584171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583708    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33d12f3d-f13d-4f39-9fd3-c77414affabc-ovnkube-script-lib\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583740    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-node-log\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583739    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-systemd-units\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583766    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583795    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-tuned\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583795    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-etc-openvswitch\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583820    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-slash\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqdkb\" (UniqueName: \"kubernetes.io/projected/33d12f3d-f13d-4f39-9fd3-c77414affabc-kube-api-access-wqdkb\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583878    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-sysctl-d\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583917    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-modprobe-d\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583920    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-sys\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583958    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-cni-bin\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.583971    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-sys\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584174    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-run-systemd\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584254    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0c1248c-5bb1-4cd3-bade-adc29339979c-dbus\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584265    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-cni-bin\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584303    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-log-socket\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.584340    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:35.585112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-var-lib-kubelet\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584431    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-slash\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584451    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584461    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-kubelet\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584479    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-run-openvswitch\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584492    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584499    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-node-log\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584497    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9191e46f-8231-4d1c-8104-c679ad0e2eeb-hosts-file\") pod \"node-resolver-qm8fd\" (UID: \"9191e46f-8231-4d1c-8104-c679ad0e2eeb\") " pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-run\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584534    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-run-netns\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584566    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-host-cni-netd\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584818    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9191e46f-8231-4d1c-8104-c679ad0e2eeb-tmp-dir\") pod \"node-resolver-qm8fd\" (UID: \"9191e46f-8231-4d1c-8104-c679ad0e2eeb\") " pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.584933    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6c5f10b-41ce-40fc-b5dd-d49bdcca7763-iptables-alerter-script\") pod \"iptables-alerter-bsvn9\" (UID: \"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763\") " pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.585001    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33d12f3d-f13d-4f39-9fd3-c77414affabc-var-lib-openvswitch\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.585020    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33d12f3d-f13d-4f39-9fd3-c77414affabc-env-overrides\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.585065    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-sysctl-d\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.585085    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret podName:b0c1248c-5bb1-4cd3-bade-adc29339979c nodeName:}" failed. No retries permitted until 2026-07-29 20:24:36.085065081 +0000 UTC m=+3.128757268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret") pod "global-pull-secret-syncer-9xh8g" (UID: "b0c1248c-5bb1-4cd3-bade-adc29339979c") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:35.586051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.585109    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33d12f3d-f13d-4f39-9fd3-c77414affabc-ovnkube-script-lib\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.586960 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.585143    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-lib-modules\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.586960 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.585517    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33d12f3d-f13d-4f39-9fd3-c77414affabc-ovnkube-config\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.587155 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.587134    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33d12f3d-f13d-4f39-9fd3-c77414affabc-ovn-node-metrics-cert\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.587709 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.587682    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-tmp\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.588683 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.588652    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-etc-tuned\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.595131 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.595039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqdkb\" (UniqueName: \"kubernetes.io/projected/33d12f3d-f13d-4f39-9fd3-c77414affabc-kube-api-access-wqdkb\") pod \"ovnkube-node-5lprd\" (UID: \"33d12f3d-f13d-4f39-9fd3-c77414affabc\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.595701 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.595659    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgtb\" (UniqueName: \"kubernetes.io/projected/a4a26b6f-575c-4748-ac8e-ed0f7a6c236a-kube-api-access-tjgtb\") pod \"tuned-29rwk\" (UID: \"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a\") " pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.596394 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.596358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps6cf\" (UniqueName: \"kubernetes.io/projected/c6c5f10b-41ce-40fc-b5dd-d49bdcca7763-kube-api-access-ps6cf\") pod \"iptables-alerter-bsvn9\" (UID: \"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763\") " pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.597966 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.597895    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4k2\" (UniqueName: \"kubernetes.io/projected/9191e46f-8231-4d1c-8104-c679ad0e2eeb-kube-api-access-mk4k2\") pod \"node-resolver-qm8fd\" (UID: \"9191e46f-8231-4d1c-8104-c679ad0e2eeb\") " pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.670235 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.670152    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7z62g"
Jul 29 20:24:35.677616 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.677595    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:35.688239 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.688076    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k58wb"
Jul 29 20:24:35.689803 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.689747    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:24:35.693497 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.693475    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c"
Jul 29 20:24:35.701497 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.701478    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6hgg4"
Jul 29 20:24:35.707111 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.706953    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qm8fd"
Jul 29 20:24:35.715943 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.715922    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-29rwk"
Jul 29 20:24:35.725994 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.725975    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bsvn9"
Jul 29 20:24:35.731973 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.731952    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:24:35.914161 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:35.914130    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26eecc01_788c_477e_b50a_5553413d3dd8.slice/crio-da9f9a0e51fe65f4a6a0c863bb5c5370b2c106d4ea1481459b8b4f1959c66287 WatchSource:0}: Error finding container da9f9a0e51fe65f4a6a0c863bb5c5370b2c106d4ea1481459b8b4f1959c66287: Status 404 returned error can't find the container with id da9f9a0e51fe65f4a6a0c863bb5c5370b2c106d4ea1481459b8b4f1959c66287
Jul 29 20:24:35.987407 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:35.987379    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:35.987546 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.987488    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:35.987546 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:35.987545    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs podName:94bf5594-165a-4efb-ba8c-a628ea34eabf nodeName:}" failed. No retries permitted until 2026-07-29 20:24:36.987528785 +0000 UTC m=+4.031220947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs") pod "network-metrics-daemon-7sqwj" (UID: "94bf5594-165a-4efb-ba8c-a628ea34eabf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:36.088205 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:36.088172    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:36.088366 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:36.088277    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:36.088444 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:36.088381    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:24:36.088444 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:36.088396    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:36.088444 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:36.088406    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:24:36.088444 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:36.088419    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ngk9r for pod openshift-network-diagnostics/network-check-target-zmnd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:36.088647 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:36.088451    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret podName:b0c1248c-5bb1-4cd3-bade-adc29339979c nodeName:}" failed. No retries permitted until 2026-07-29 20:24:37.088431401 +0000 UTC m=+4.132123565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret") pod "global-pull-secret-syncer-9xh8g" (UID: "b0c1248c-5bb1-4cd3-bade-adc29339979c") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:36.088647 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:36.088491    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r podName:a0d385ca-876f-4748-a57c-8e0c3f5ca7cc nodeName:}" failed. No retries permitted until 2026-07-29 20:24:37.088474171 +0000 UTC m=+4.132166338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngk9r" (UniqueName: "kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r") pod "network-check-target-zmnd7" (UID: "a0d385ca-876f-4748-a57c-8e0c3f5ca7cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:36.412326 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:36.412283    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:19:35 +0000 UTC" deadline="2028-01-21 06:25:12.773751847 +0000 UTC"
Jul 29 20:24:36.412326 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:36.412321    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12970h0m36.361436286s"
Jul 29 20:24:36.518346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:36.518261    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:36.518496 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:36.518401    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:36.528747 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:36.528715    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h9t4t" event={"ID":"26eecc01-788c-477e-b50a-5553413d3dd8","Type":"ContainerStarted","Data":"da9f9a0e51fe65f4a6a0c863bb5c5370b2c106d4ea1481459b8b4f1959c66287"}
Jul 29 20:24:36.590338 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:36.590306    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf47463_470a_4c86_8301_a838734a4963.slice/crio-de920c0d8768f938aceec4299edadcb1181c8483c59be52af0a90155ea633dfe WatchSource:0}: Error finding container de920c0d8768f938aceec4299edadcb1181c8483c59be52af0a90155ea633dfe: Status 404 returned error can't find the container with id de920c0d8768f938aceec4299edadcb1181c8483c59be52af0a90155ea633dfe
Jul 29 20:24:36.591282 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:36.591172    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a2a86a_79da_4c24_b0e1_0a3a3b510b73.slice/crio-ce3679f7f52c9e41edc7f1b695b4ee6924bb956e89aed1a60331b9e3abc67334 WatchSource:0}: Error finding container ce3679f7f52c9e41edc7f1b695b4ee6924bb956e89aed1a60331b9e3abc67334: Status 404 returned error can't find the container with id ce3679f7f52c9e41edc7f1b695b4ee6924bb956e89aed1a60331b9e3abc67334
Jul 29 20:24:36.593078 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:36.592982    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a26b6f_575c_4748_ac8e_ed0f7a6c236a.slice/crio-2026fea0d8593e8f30c5459802574028781e4220687ef42ad99b97a5e6f2579f WatchSource:0}: Error finding container 2026fea0d8593e8f30c5459802574028781e4220687ef42ad99b97a5e6f2579f: Status 404 returned error can't find the container with id 2026fea0d8593e8f30c5459802574028781e4220687ef42ad99b97a5e6f2579f
Jul 29 20:24:36.598997 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:36.598974    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f01f25d_0021_40f3_a536_01a921d1557d.slice/crio-36b7780f38633139efa1bca96b11cd05eabea3c62b453841b5cb5c3c05b66077 WatchSource:0}: Error finding container 36b7780f38633139efa1bca96b11cd05eabea3c62b453841b5cb5c3c05b66077: Status 404 returned error can't find the container with id 36b7780f38633139efa1bca96b11cd05eabea3c62b453841b5cb5c3c05b66077
Jul 29 20:24:36.600000 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:36.599972    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9191e46f_8231_4d1c_8104_c679ad0e2eeb.slice/crio-dc54a852404ce7f1151c24a5e4177c90aace5e86db0482d87b564cf7f5d22652 WatchSource:0}: Error finding container dc54a852404ce7f1151c24a5e4177c90aace5e86db0482d87b564cf7f5d22652: Status 404 returned error can't find the container with id dc54a852404ce7f1151c24a5e4177c90aace5e86db0482d87b564cf7f5d22652
Jul 29 20:24:36.601382 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:36.601164    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d12f3d_f13d_4f39_9fd3_c77414affabc.slice/crio-2b26e2bac06d454cf7229b587ba9d4ebb646c6bc6d15819a8380dfa17cc5d5f1 WatchSource:0}: Error finding container 2b26e2bac06d454cf7229b587ba9d4ebb646c6bc6d15819a8380dfa17cc5d5f1: Status 404 returned error can't find the container with id 2b26e2bac06d454cf7229b587ba9d4ebb646c6bc6d15819a8380dfa17cc5d5f1
Jul 29 20:24:36.605662 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:24:36.605640    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78b2c7b_217a_4490_921d_84357434b580.slice/crio-d361c8bfae01f0ba88db8e175ada58c2ce3c5b976bb06db99d810732715dc40c WatchSource:0}: Error finding container d361c8bfae01f0ba88db8e175ada58c2ce3c5b976bb06db99d810732715dc40c: Status 404 returned error can't find the container with id d361c8bfae01f0ba88db8e175ada58c2ce3c5b976bb06db99d810732715dc40c
Jul 29 20:24:36.995063 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:36.994757    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:36.995063 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:36.994906    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:36.995284 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:36.995102    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs podName:94bf5594-165a-4efb-ba8c-a628ea34eabf nodeName:}" failed. No retries permitted until 2026-07-29 20:24:38.995068568 +0000 UTC m=+6.038760745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs") pod "network-metrics-daemon-7sqwj" (UID: "94bf5594-165a-4efb-ba8c-a628ea34eabf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:37.095900 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.095822    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:37.095900 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.095881    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:37.096114 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:37.096027    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:24:37.096114 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:37.096049    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:24:37.096114 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:37.096062    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ngk9r for pod openshift-network-diagnostics/network-check-target-zmnd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:37.096272 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:37.096119    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r podName:a0d385ca-876f-4748-a57c-8e0c3f5ca7cc nodeName:}" failed. No retries permitted until 2026-07-29 20:24:39.096098908 +0000 UTC m=+6.139791073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngk9r" (UniqueName: "kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r") pod "network-check-target-zmnd7" (UID: "a0d385ca-876f-4748-a57c-8e0c3f5ca7cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:37.096542 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:37.096522    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:37.096615 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:37.096585    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret podName:b0c1248c-5bb1-4cd3-bade-adc29339979c nodeName:}" failed. No retries permitted until 2026-07-29 20:24:39.096565057 +0000 UTC m=+6.140257221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret") pod "global-pull-secret-syncer-9xh8g" (UID: "b0c1248c-5bb1-4cd3-bade-adc29339979c") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:37.412709 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.412566    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:19:35 +0000 UTC" deadline="2028-03-05 10:29:05.551276325 +0000 UTC"
Jul 29 20:24:37.412709 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.412627    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14030h4m28.138654465s"
Jul 29 20:24:37.520630 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.518071    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:37.520630 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:37.518210    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:37.520630 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.518276    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:37.520630 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:37.518545    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:37.549729 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.549695    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7z62g" event={"ID":"a78b2c7b-217a-4490-921d-84357434b580","Type":"ContainerStarted","Data":"d361c8bfae01f0ba88db8e175ada58c2ce3c5b976bb06db99d810732715dc40c"}
Jul 29 20:24:37.562471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.557859    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6hgg4" event={"ID":"1f01f25d-0021-40f3-a536-01a921d1557d","Type":"ContainerStarted","Data":"36b7780f38633139efa1bca96b11cd05eabea3c62b453841b5cb5c3c05b66077"}
Jul 29 20:24:37.569772 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.569624    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qm8fd" event={"ID":"9191e46f-8231-4d1c-8104-c679ad0e2eeb","Type":"ContainerStarted","Data":"dc54a852404ce7f1151c24a5e4177c90aace5e86db0482d87b564cf7f5d22652"}
Jul 29 20:24:37.586215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.586122    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" event={"ID":"33d12f3d-f13d-4f39-9fd3-c77414affabc","Type":"ContainerStarted","Data":"2b26e2bac06d454cf7229b587ba9d4ebb646c6bc6d15819a8380dfa17cc5d5f1"}
Jul 29 20:24:37.601556 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.601476    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-29rwk" event={"ID":"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a","Type":"ContainerStarted","Data":"2026fea0d8593e8f30c5459802574028781e4220687ef42ad99b97a5e6f2579f"}
Jul 29 20:24:37.619264 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.619229    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c" event={"ID":"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73","Type":"ContainerStarted","Data":"ce3679f7f52c9e41edc7f1b695b4ee6924bb956e89aed1a60331b9e3abc67334"}
Jul 29 20:24:37.635652 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.635558    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" event={"ID":"1f03ef72ad08cd4666542db4e9e3bb71","Type":"ContainerStarted","Data":"6efc8c19bfdbd304a2c55b6eb865a1aa5ccc410d605ec0d5523bc09139dae0d6"}
Jul 29 20:24:37.654790 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.654757    2585 generic.go:356] "Generic (PLEG): container finished" podID="0d6135b471ad91bdc496cb5531f8ee37" containerID="07af2948725668fc9e037a632da3cea7e446850b79ea1b327c522d30c55d9952" exitCode=0
Jul 29 20:24:37.654908 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.654875    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" event={"ID":"0d6135b471ad91bdc496cb5531f8ee37","Type":"ContainerDied","Data":"07af2948725668fc9e037a632da3cea7e446850b79ea1b327c522d30c55d9952"}
Jul 29 20:24:37.670511 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.670447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bsvn9" event={"ID":"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763","Type":"ContainerStarted","Data":"c005f0a5c4273244afb472ab1c23ebc3d5a77825068a763e740d5db5e46e90b2"}
Jul 29 20:24:37.676393 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.675782    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" podStartSLOduration=2.675762933 podStartE2EDuration="2.675762933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:24:37.656972384 +0000 UTC m=+4.700664573" watchObservedRunningTime="2026-07-29 20:24:37.675762933 +0000 UTC m=+4.719455123"
Jul 29 20:24:37.683488 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:37.683401    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k58wb" event={"ID":"7cf47463-470a-4c86-8301-a838734a4963","Type":"ContainerStarted","Data":"de920c0d8768f938aceec4299edadcb1181c8483c59be52af0a90155ea633dfe"}
Jul 29 20:24:38.518802 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:38.518768    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:38.519487 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:38.518904    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:38.700860 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:38.700464    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" event={"ID":"0d6135b471ad91bdc496cb5531f8ee37","Type":"ContainerStarted","Data":"b9ff097b9ec35ed34168ff3a5205af25c5d043619d0616a7c2f2df4bc14bb654"}
Jul 29 20:24:38.718406 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:38.718355    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" podStartSLOduration=3.718333915 podStartE2EDuration="3.718333915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:24:38.71809177 +0000 UTC m=+5.761783957" watchObservedRunningTime="2026-07-29 20:24:38.718333915 +0000 UTC m=+5.762026105"
Jul 29 20:24:39.015122 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:39.015002    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:39.015311 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.015168    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:39.015311 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.015239    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs podName:94bf5594-165a-4efb-ba8c-a628ea34eabf nodeName:}" failed. No retries permitted until 2026-07-29 20:24:43.015218053 +0000 UTC m=+10.058910240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs") pod "network-metrics-daemon-7sqwj" (UID: "94bf5594-165a-4efb-ba8c-a628ea34eabf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:39.116713 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:39.115797    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:39.116713 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:39.115864    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:39.116713 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.116059    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:24:39.116713 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.116080    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:24:39.116713 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.116094    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ngk9r for pod openshift-network-diagnostics/network-check-target-zmnd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:39.116713 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.116153    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r podName:a0d385ca-876f-4748-a57c-8e0c3f5ca7cc nodeName:}" failed. No retries permitted until 2026-07-29 20:24:43.116135511 +0000 UTC m=+10.159827689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngk9r" (UniqueName: "kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r") pod "network-check-target-zmnd7" (UID: "a0d385ca-876f-4748-a57c-8e0c3f5ca7cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:39.116713 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.116600    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:39.116713 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.116666    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret podName:b0c1248c-5bb1-4cd3-bade-adc29339979c nodeName:}" failed. No retries permitted until 2026-07-29 20:24:43.116649967 +0000 UTC m=+10.160342135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret") pod "global-pull-secret-syncer-9xh8g" (UID: "b0c1248c-5bb1-4cd3-bade-adc29339979c") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:39.520333 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:39.519508    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:39.520333 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.519639    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:39.521100 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:39.520933    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:39.521100 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:39.521051    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:40.519093 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:40.518585    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:40.519093 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:40.518723    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:41.520949 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:41.520449    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:41.520949 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:41.520582    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:41.520949 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:41.520705    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:41.520949 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:41.520832    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:42.518827 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:42.518754    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:42.518983 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:42.518879    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:43.051616 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:43.051581    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:43.052032 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.051728    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:43.052032 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.051786    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs podName:94bf5594-165a-4efb-ba8c-a628ea34eabf nodeName:}" failed. No retries permitted until 2026-07-29 20:24:51.051766474 +0000 UTC m=+18.095458636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs") pod "network-metrics-daemon-7sqwj" (UID: "94bf5594-165a-4efb-ba8c-a628ea34eabf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:43.152817 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:43.152747    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:43.152991 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:43.152877    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:43.152991 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.152937    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:24:43.152991 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.152962    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:24:43.152991 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.152977    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ngk9r for pod openshift-network-diagnostics/network-check-target-zmnd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:43.152991 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.152979    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:43.153246 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.153037    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret podName:b0c1248c-5bb1-4cd3-bade-adc29339979c nodeName:}" failed. No retries permitted until 2026-07-29 20:24:51.153020613 +0000 UTC m=+18.196712788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret") pod "global-pull-secret-syncer-9xh8g" (UID: "b0c1248c-5bb1-4cd3-bade-adc29339979c") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:43.153246 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.153056    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r podName:a0d385ca-876f-4748-a57c-8e0c3f5ca7cc nodeName:}" failed. No retries permitted until 2026-07-29 20:24:51.153048069 +0000 UTC m=+18.196740233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngk9r" (UniqueName: "kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r") pod "network-check-target-zmnd7" (UID: "a0d385ca-876f-4748-a57c-8e0c3f5ca7cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:43.519363 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:43.518852    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:43.519363 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.518974    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:43.519363 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:43.519040    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:43.519363 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:43.519132    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:44.518722 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:44.518680    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:44.519284 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:44.518811    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:45.518905 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:45.518873    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:45.519382 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:45.519006    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:45.519382 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:45.519160    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:45.519382 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:45.519271    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:46.518259 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:46.518222    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:46.518429 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:46.518371    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:47.518233 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:47.518200    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:47.518725 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:47.518317    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:47.518725 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:47.518403    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:47.518725 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:47.518537    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:48.518123 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:48.518099    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:48.518297 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:48.518208    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:49.518779 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:49.518741    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:49.519229 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:49.518859    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:49.519229 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:49.518920    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:49.519229 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:49.519043    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:50.518311 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:50.518279    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:50.518585 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:50.518404    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:51.115992 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:51.115956    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:51.116410 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.116116    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:51.116410 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.116186    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs podName:94bf5594-165a-4efb-ba8c-a628ea34eabf nodeName:}" failed. No retries permitted until 2026-07-29 20:25:07.116164813 +0000 UTC m=+34.159856974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs") pod "network-metrics-daemon-7sqwj" (UID: "94bf5594-165a-4efb-ba8c-a628ea34eabf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:24:51.216724 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:51.216699    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:51.216857 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:51.216759    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:51.216857 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.216850    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:51.216952 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.216866    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:24:51.216952 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.216882    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:24:51.216952 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.216894    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ngk9r for pod openshift-network-diagnostics/network-check-target-zmnd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:51.216952 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.216908    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret podName:b0c1248c-5bb1-4cd3-bade-adc29339979c nodeName:}" failed. No retries permitted until 2026-07-29 20:25:07.216890166 +0000 UTC m=+34.260582336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret") pod "global-pull-secret-syncer-9xh8g" (UID: "b0c1248c-5bb1-4cd3-bade-adc29339979c") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:24:51.216952 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.216928    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r podName:a0d385ca-876f-4748-a57c-8e0c3f5ca7cc nodeName:}" failed. No retries permitted until 2026-07-29 20:25:07.216918328 +0000 UTC m=+34.260610493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngk9r" (UniqueName: "kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r") pod "network-check-target-zmnd7" (UID: "a0d385ca-876f-4748-a57c-8e0c3f5ca7cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:24:51.518573 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:51.518536    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:51.518718 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.518651    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:51.518718 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:51.518703    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:51.518815 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:51.518796    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:52.518285 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:52.518253    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:52.518694 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:52.518374    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:53.518847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:53.518811    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:53.519245 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:53.518940    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:53.519245 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:53.518989    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:53.519245 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:53.519081    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:54.518703 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:54.518676    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:54.518868 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:54.518791    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:55.519289 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.519096    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:55.519722 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:55.519405    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:55.519722 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.519480    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:55.519722 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:55.519621    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:55.735898 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.735729    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k58wb" event={"ID":"7cf47463-470a-4c86-8301-a838734a4963","Type":"ContainerStarted","Data":"0d93bc57446b135a995af39c651f763e320decba1a97ad1b92ad589769483839"}
Jul 29 20:24:55.737252 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.737230    2585 generic.go:356] "Generic (PLEG): container finished" podID="a78b2c7b-217a-4490-921d-84357434b580" containerID="52cfba133e8c7ef029cb8a3c0e08c1e3613f4a0e9767bfdcc6bffc41a398327a" exitCode=0
Jul 29 20:24:55.737391 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.737296    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7z62g" event={"ID":"a78b2c7b-217a-4490-921d-84357434b580","Type":"ContainerDied","Data":"52cfba133e8c7ef029cb8a3c0e08c1e3613f4a0e9767bfdcc6bffc41a398327a"}
Jul 29 20:24:55.739140 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.738993    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6hgg4" event={"ID":"1f01f25d-0021-40f3-a536-01a921d1557d","Type":"ContainerStarted","Data":"077b89414708682224e2664b3bae8782fa8261f5bbfe77cff9e21feb4c0d0121"}
Jul 29 20:24:55.740428 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.740401    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qm8fd" event={"ID":"9191e46f-8231-4d1c-8104-c679ad0e2eeb","Type":"ContainerStarted","Data":"1ad2afd958d7e4ae1e8efaabd5bc7aea86b22c957fc25d371025a8991390211c"}
Jul 29 20:24:55.742270 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.742249    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" event={"ID":"33d12f3d-f13d-4f39-9fd3-c77414affabc","Type":"ContainerStarted","Data":"bf37e556b6b88ce61479750a2d75c7dc51624a6e101498260a82c06f71f5562d"}
Jul 29 20:24:55.742406 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.742302    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" event={"ID":"33d12f3d-f13d-4f39-9fd3-c77414affabc","Type":"ContainerStarted","Data":"52773e9d2dffec7cec2b8af4e07e10c23bbf98a917edf4066b462cc78fa8dc30"}
Jul 29 20:24:55.743530 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.743506    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-29rwk" event={"ID":"a4a26b6f-575c-4748-ac8e-ed0f7a6c236a","Type":"ContainerStarted","Data":"ffaa3e6919a19f493358e5f84b2a5feaf893ecf9e865ef943e00e94f5bd175d1"}
Jul 29 20:24:55.744924 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.744895    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c" event={"ID":"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73","Type":"ContainerStarted","Data":"83723624117bc095712a9a6b5a542da2106036fa74b8cda1394eba95a0c4f94a"}
Jul 29 20:24:55.746607 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.746587    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h9t4t" event={"ID":"26eecc01-788c-477e-b50a-5553413d3dd8","Type":"ContainerStarted","Data":"c31e9d159e823ec7fe39e07581918fdb54dfe4812cffd00020000451bbc1dc95"}
Jul 29 20:24:55.752826 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.752789    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k58wb" podStartSLOduration=3.009133547 podStartE2EDuration="21.752776511s" totalImagesPullingTime="18.743642964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:24:55.75225611 +0000 UTC m=+22.795948293" watchObservedRunningTime="2026-07-29 20:24:55.752776511 +0000 UTC m=+22.796468712"
Jul 29 20:24:55.767202 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.767097    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qm8fd" podStartSLOduration=3.058154316 podStartE2EDuration="21.767083712s" totalImagesPullingTime="18.708929396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:24:55.766653409 +0000 UTC m=+22.810345605" watchObservedRunningTime="2026-07-29 20:24:55.767083712 +0000 UTC m=+22.810775895"
Jul 29 20:24:55.787859 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.787819    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-29rwk" podStartSLOduration=3.0456506819999998 podStartE2EDuration="21.787805755s" totalImagesPullingTime="18.742155073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:24:55.787697747 +0000 UTC m=+22.831389932" watchObservedRunningTime="2026-07-29 20:24:55.787805755 +0000 UTC m=+22.831497960"
Jul 29 20:24:55.804110 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.804068    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h9t4t" podStartSLOduration=2.46410062 podStartE2EDuration="21.80405376s" totalImagesPullingTime="19.33995314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:24:55.803807107 +0000 UTC m=+22.847499292" watchObservedRunningTime="2026-07-29 20:24:55.80405376 +0000 UTC m=+22.847745943"
Jul 29 20:24:55.825633 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:55.825595    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6hgg4" podStartSLOduration=2.913256157 podStartE2EDuration="21.825579755s" totalImagesPullingTime="18.912323598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:24:55.825471643 +0000 UTC m=+22.869163826" watchObservedRunningTime="2026-07-29 20:24:55.825579755 +0000 UTC m=+22.869271939"
Jul 29 20:24:56.518105 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:56.518073    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:56.518397 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:56.518191    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:56.752539 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:56.752488    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" event={"ID":"33d12f3d-f13d-4f39-9fd3-c77414affabc","Type":"ContainerStarted","Data":"aad08941d227c40ac0a9872b5f097a70be2f3925de434d164ed67699fb9923da"}
Jul 29 20:24:56.753351 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:56.752579    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" event={"ID":"33d12f3d-f13d-4f39-9fd3-c77414affabc","Type":"ContainerStarted","Data":"69db67c538d0cd6f1b36cb95e3d4897798f229ae79e5571009f5c56ac195ee80"}
Jul 29 20:24:56.753351 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:56.752628    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" event={"ID":"33d12f3d-f13d-4f39-9fd3-c77414affabc","Type":"ContainerStarted","Data":"f344073b8fcbc50d84526f069056883694661219311535d2cfebd80b1ec83bbe"}
Jul 29 20:24:56.753351 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:56.752683    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" event={"ID":"33d12f3d-f13d-4f39-9fd3-c77414affabc","Type":"ContainerStarted","Data":"06989e515d283fa9c4d8b02a19a2308a97e9d3aa599926b12e9bb4ce5004f2b9"}
Jul 29 20:24:56.770316 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:56.770289    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 20:24:57.110776 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.110738    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:57.111487 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.111466    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:57.426527 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.426360    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T20:24:56.770304403Z","UUID":"ce6c5ca9-8fcf-4fcc-a5b8-fd586be5bc40","Handler":null,"Name":"","Endpoint":""}
Jul 29 20:24:57.427901 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.427860    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 20:24:57.427901 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.427902    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 20:24:57.518085 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.518053    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:57.518085 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.518065    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:57.518294 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:57.518177    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:57.518294 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:57.518259    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:24:57.756905 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.756874    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c" event={"ID":"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73","Type":"ContainerStarted","Data":"5b8e4c718b2e5b61d2fc85d747bac08af1a9bcadc53095d25f437ead6ff4e676"}
Jul 29 20:24:57.758742 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.758708    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bsvn9" event={"ID":"c6c5f10b-41ce-40fc-b5dd-d49bdcca7763","Type":"ContainerStarted","Data":"f8cf371a08c45536135119ac76eccf5718c28830812548eacf3f7abb631c1bba"}
Jul 29 20:24:57.759187 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.759163    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:57.759643 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.759627    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h9t4t"
Jul 29 20:24:57.773762 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:57.773728    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bsvn9" podStartSLOduration=5.027609951 podStartE2EDuration="23.77371603s" totalImagesPullingTime="18.746106079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:24:57.77314308 +0000 UTC m=+24.816835262" watchObservedRunningTime="2026-07-29 20:24:57.77371603 +0000 UTC m=+24.817408212"
Jul 29 20:24:58.518510 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:58.518341    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:24:58.518685 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:58.518602    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:24:58.765321 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:58.765283    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" event={"ID":"33d12f3d-f13d-4f39-9fd3-c77414affabc","Type":"ContainerStarted","Data":"b7982ebb846dbab20b0dbd74964dc5e5144e16c2f329ead4edceec7bc1987a68"}
Jul 29 20:24:58.767655 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:58.767623    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c" event={"ID":"e5a2a86a-79da-4c24-b0e1-0a3a3b510b73","Type":"ContainerStarted","Data":"27c97002b7e3473d23fb173bbc6824a0382f758c2c0cfb31fe9ef170d541383a"}
Jul 29 20:24:58.786347 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:58.786269    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2r4c" podStartSLOduration=3.803423337 podStartE2EDuration="24.786254329s" totalImagesPullingTime="20.982830992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:24:58.786090043 +0000 UTC m=+25.829782226" watchObservedRunningTime="2026-07-29 20:24:58.786254329 +0000 UTC m=+25.829946512"
Jul 29 20:24:59.518897 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:59.518858    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:24:59.519091 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:24:59.518864    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:24:59.519091 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:59.518976    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:24:59.519091 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:24:59.519047    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:25:00.518468 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.518234    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:00.519064 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:00.518578    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:25:00.773576 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.773548    2585 generic.go:356] "Generic (PLEG): container finished" podID="a78b2c7b-217a-4490-921d-84357434b580" containerID="b4b8b0d8d4d3421bf7057f7da99c3202e6c6453efabb6f0eefda62b5f5a1b0c9" exitCode=0
Jul 29 20:25:00.773743 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.773627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7z62g" event={"ID":"a78b2c7b-217a-4490-921d-84357434b580","Type":"ContainerDied","Data":"b4b8b0d8d4d3421bf7057f7da99c3202e6c6453efabb6f0eefda62b5f5a1b0c9"}
Jul 29 20:25:00.777032 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.776993    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" event={"ID":"33d12f3d-f13d-4f39-9fd3-c77414affabc","Type":"ContainerStarted","Data":"0fd4741784e3350330cf11e16fedc7f13757f252d526c656d111bf8d9c221ff0"}
Jul 29 20:25:00.777324 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.777307    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:25:00.777427 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.777397    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:25:00.777501 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.777447    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:25:00.793171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.793151    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:25:00.797708 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.797691    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:25:00.829711 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:00.829674    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd" podStartSLOduration=7.9373074500000005 podStartE2EDuration="26.829661047s" totalImagesPullingTime="18.892353597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:25:00.828011026 +0000 UTC m=+27.871703209" watchObservedRunningTime="2026-07-29 20:25:00.829661047 +0000 UTC m=+27.873353229"
Jul 29 20:25:01.518264 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:01.518232    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:01.518495 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:01.518242    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:25:01.518495 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:01.518389    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:25:01.518495 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:01.518448    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:25:02.213611 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:02.213580    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zmnd7"]
Jul 29 20:25:02.213789 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:02.213702    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:02.213853 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:02.213800    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:25:02.216509 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:02.216487    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9xh8g"]
Jul 29 20:25:02.216619 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:02.216557    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:02.216659 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:02.216620    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:25:02.219230 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:02.219207    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7sqwj"]
Jul 29 20:25:02.219367 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:02.219287    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:25:02.219420 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:02.219382    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:25:02.783115 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:02.783078    2585 generic.go:356] "Generic (PLEG): container finished" podID="a78b2c7b-217a-4490-921d-84357434b580" containerID="2f33c7344dae9048afbf716fd83461589831fd7f690b532a10433f8541b477cf" exitCode=0
Jul 29 20:25:02.783565 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:02.783154    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7z62g" event={"ID":"a78b2c7b-217a-4490-921d-84357434b580","Type":"ContainerDied","Data":"2f33c7344dae9048afbf716fd83461589831fd7f690b532a10433f8541b477cf"}
Jul 29 20:25:03.520041 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:03.520001    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:03.520220 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:03.520147    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:25:04.518896 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:04.518730    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:25:04.519290 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:04.518783    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:04.519290 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:04.518969    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:25:04.519290 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:04.519065    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:25:04.789946 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:04.789908    2585 generic.go:356] "Generic (PLEG): container finished" podID="a78b2c7b-217a-4490-921d-84357434b580" containerID="9b2a26f8615828abf30a150e553fcc361703d94e894e6f37234e95aff994530c" exitCode=0
Jul 29 20:25:04.790078 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:04.789954    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7z62g" event={"ID":"a78b2c7b-217a-4490-921d-84357434b580","Type":"ContainerDied","Data":"9b2a26f8615828abf30a150e553fcc361703d94e894e6f37234e95aff994530c"}
Jul 29 20:25:05.517990 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:05.517960    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:05.518194 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:05.518089    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:25:06.518686 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:06.518650    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:25:06.519216 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:06.518785    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqwj" podUID="94bf5594-165a-4efb-ba8c-a628ea34eabf"
Jul 29 20:25:06.519216 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:06.518838    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:06.519216 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:06.518955    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmnd7" podUID="a0d385ca-876f-4748-a57c-8e0c3f5ca7cc"
Jul 29 20:25:07.142615 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:07.142576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:25:07.142794 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:07.142726    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:25:07.142794 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:07.142790    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs podName:94bf5594-165a-4efb-ba8c-a628ea34eabf nodeName:}" failed. No retries permitted until 2026-07-29 20:25:39.14277059 +0000 UTC m=+66.186462765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs") pod "network-metrics-daemon-7sqwj" (UID: "94bf5594-165a-4efb-ba8c-a628ea34eabf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:25:07.243435 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:07.243402    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:07.243591 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:07.243477    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:07.243591 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:07.243562    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:25:07.243591 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:07.243564    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:25:07.243591 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:07.243588    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:25:07.243738 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:07.243599    2585 projected.go:196] Error preparing data for projected volume kube-api-access-ngk9r for pod openshift-network-diagnostics/network-check-target-zmnd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:25:07.243738 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:07.243612    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret podName:b0c1248c-5bb1-4cd3-bade-adc29339979c nodeName:}" failed. No retries permitted until 2026-07-29 20:25:39.243599981 +0000 UTC m=+66.287292142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret") pod "global-pull-secret-syncer-9xh8g" (UID: "b0c1248c-5bb1-4cd3-bade-adc29339979c") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:25:07.243738 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:07.243635    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r podName:a0d385ca-876f-4748-a57c-8e0c3f5ca7cc nodeName:}" failed. No retries permitted until 2026-07-29 20:25:39.243624269 +0000 UTC m=+66.287316430 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ngk9r" (UniqueName: "kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r") pod "network-check-target-zmnd7" (UID: "a0d385ca-876f-4748-a57c-8e0c3f5ca7cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:25:07.518218 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:07.518182    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:07.518417 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:07.518323    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xh8g" podUID="b0c1248c-5bb1-4cd3-bade-adc29339979c"
Jul 29 20:25:08.310232 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.310207    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeReady"
Jul 29 20:25:08.310766 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.310386    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 20:25:08.353088 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.353054    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"]
Jul 29 20:25:08.355709 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.355686    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.359648 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.359628    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 20:25:08.359997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.359979    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-brdv4\""
Jul 29 20:25:08.360107 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.360080    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 20:25:08.360709 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.360687    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 20:25:08.365952 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.365928    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 20:25:08.366946 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.366925    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cgmsd"]
Jul 29 20:25:08.368805 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.368788    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.371429 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.371278    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 20:25:08.371639 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.371624    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xpqnl\""
Jul 29 20:25:08.371751 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.371734    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 20:25:08.373741 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.373722    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"]
Jul 29 20:25:08.376471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.376249    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-knr4r"]
Jul 29 20:25:08.378533 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.378512    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-knr4r"
Jul 29 20:25:08.381725 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.381542    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 20:25:08.381725 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.381641    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cs4sn\""
Jul 29 20:25:08.381725 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.381679    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 20:25:08.382951 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.382899    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 20:25:08.384069 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.384009    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cgmsd"]
Jul 29 20:25:08.389883 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.389862    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-knr4r"]
Jul 29 20:25:08.518147 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.518112    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:08.518147 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.518153    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:25:08.522476 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.522434    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 20:25:08.522476 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.522451    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mkp42\""
Jul 29 20:25:08.522688 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.522449    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 20:25:08.522688 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.522587    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7tqs\""
Jul 29 20:25:08.522688 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.522621    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 20:25:08.548377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.548281    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:08.553202 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553175    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48bb1c6c-0d0e-436c-af8d-747647703600-cert\") pod \"ingress-canary-knr4r\" (UID: \"48bb1c6c-0d0e-436c-af8d-747647703600\") " pod="openshift-ingress-canary/ingress-canary-knr4r"
Jul 29 20:25:08.553311 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553219    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2883174b-258b-4a7b-906d-237dc51e8027-ca-trust-extracted\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.553311 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553245    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-registry-certificates\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.553311 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553300    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.553501 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553392    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl4lv\" (UniqueName: \"kubernetes.io/projected/48bb1c6c-0d0e-436c-af8d-747647703600-kube-api-access-vl4lv\") pod \"ingress-canary-knr4r\" (UID: \"48bb1c6c-0d0e-436c-af8d-747647703600\") " pod="openshift-ingress-canary/ingress-canary-knr4r"
Jul 29 20:25:08.553501 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553429    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.553501 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553453    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c106d721-0381-4f51-ad74-dcb1804221fc-tmp-dir\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.553501 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553488    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-installation-pull-secrets\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.553707 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553532    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c106d721-0381-4f51-ad74-dcb1804221fc-config-volume\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.553707 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553603    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw4tx\" (UniqueName: \"kubernetes.io/projected/c106d721-0381-4f51-ad74-dcb1804221fc-kube-api-access-bw4tx\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.553707 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553644    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-image-registry-private-configuration\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.553707 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44fnc\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-kube-api-access-44fnc\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.553880 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553715    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-trusted-ca\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.553880 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.553742    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-bound-sa-token\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.645686 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.645586    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:08.654498 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.654475    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-installation-pull-secrets\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.654614 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.654521    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c106d721-0381-4f51-ad74-dcb1804221fc-config-volume\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.654614 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.654559    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw4tx\" (UniqueName: \"kubernetes.io/projected/c106d721-0381-4f51-ad74-dcb1804221fc-kube-api-access-bw4tx\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.654614 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.654587    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-image-registry-private-configuration\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.654777 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.654617    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fnc\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-kube-api-access-44fnc\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.654777 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.654723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-trusted-ca\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.654777 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.654750    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-bound-sa-token\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.654923 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.654795    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48bb1c6c-0d0e-436c-af8d-747647703600-cert\") pod \"ingress-canary-knr4r\" (UID: \"48bb1c6c-0d0e-436c-af8d-747647703600\") " pod="openshift-ingress-canary/ingress-canary-knr4r"
Jul 29 20:25:08.654923 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.654826    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2883174b-258b-4a7b-906d-237dc51e8027-ca-trust-extracted\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.655179 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655153    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-registry-certificates\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.655265 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655194    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2883174b-258b-4a7b-906d-237dc51e8027-ca-trust-extracted\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.655265 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655196    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.655404 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:08.655265    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:25:08.655404 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655275    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl4lv\" (UniqueName: \"kubernetes.io/projected/48bb1c6c-0d0e-436c-af8d-747647703600-kube-api-access-vl4lv\") pod \"ingress-canary-knr4r\" (UID: \"48bb1c6c-0d0e-436c-af8d-747647703600\") " pod="openshift-ingress-canary/ingress-canary-knr4r"
Jul 29 20:25:08.655404 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655275    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c106d721-0381-4f51-ad74-dcb1804221fc-config-volume\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.655404 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:08.655323    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls podName:c106d721-0381-4f51-ad74-dcb1804221fc nodeName:}" failed. No retries permitted until 2026-07-29 20:25:09.155303301 +0000 UTC m=+36.198995466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls") pod "dns-default-cgmsd" (UID: "c106d721-0381-4f51-ad74-dcb1804221fc") : secret "dns-default-metrics-tls" not found
Jul 29 20:25:08.655404 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655361    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.655404 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655387    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c106d721-0381-4f51-ad74-dcb1804221fc-tmp-dir\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.655688 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655652    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c106d721-0381-4f51-ad74-dcb1804221fc-tmp-dir\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.655688 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:08.655663    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:25:08.655688 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655672    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-registry-certificates\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.655820 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:08.655681    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf8fdf9cc-85tqw: secret "image-registry-tls" not found
Jul 29 20:25:08.655820 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:08.655757    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls podName:2883174b-258b-4a7b-906d-237dc51e8027 nodeName:}" failed. No retries permitted until 2026-07-29 20:25:09.155738634 +0000 UTC m=+36.199430811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls") pod "image-registry-6bf8fdf9cc-85tqw" (UID: "2883174b-258b-4a7b-906d-237dc51e8027") : secret "image-registry-tls" not found
Jul 29 20:25:08.655820 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.655773    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-trusted-ca\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.659391 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.659369    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-installation-pull-secrets\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.659490 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.659408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-image-registry-private-configuration\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.659490 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.659422    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48bb1c6c-0d0e-436c-af8d-747647703600-cert\") pod \"ingress-canary-knr4r\" (UID: \"48bb1c6c-0d0e-436c-af8d-747647703600\") " pod="openshift-ingress-canary/ingress-canary-knr4r"
Jul 29 20:25:08.664301 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.664257    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw4tx\" (UniqueName: \"kubernetes.io/projected/c106d721-0381-4f51-ad74-dcb1804221fc-kube-api-access-bw4tx\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:08.664583 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.664563    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44fnc\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-kube-api-access-44fnc\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.664937 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.664910    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-bound-sa-token\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:08.665376 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.665311    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl4lv\" (UniqueName: \"kubernetes.io/projected/48bb1c6c-0d0e-436c-af8d-747647703600-kube-api-access-vl4lv\") pod \"ingress-canary-knr4r\" (UID: \"48bb1c6c-0d0e-436c-af8d-747647703600\") " pod="openshift-ingress-canary/ingress-canary-knr4r"
Jul 29 20:25:08.689309 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.689285    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-knr4r"
Jul 29 20:25:08.865212 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:08.865189    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-knr4r"]
Jul 29 20:25:08.869125 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:25:08.869101    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bb1c6c_0d0e_436c_af8d_747647703600.slice/crio-f693653d02c8675d7419ab255c4196b20fa52748db3f8d344d9ac348948f217b WatchSource:0}: Error finding container f693653d02c8675d7419ab255c4196b20fa52748db3f8d344d9ac348948f217b: Status 404 returned error can't find the container with id f693653d02c8675d7419ab255c4196b20fa52748db3f8d344d9ac348948f217b
Jul 29 20:25:09.159697 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:09.159660    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:09.159882 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:09.159732    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:09.159882 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:09.159785    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:25:09.159882 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:09.159807    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf8fdf9cc-85tqw: secret "image-registry-tls" not found
Jul 29 20:25:09.159882 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:09.159815    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:25:09.159882 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:09.159863    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls podName:c106d721-0381-4f51-ad74-dcb1804221fc nodeName:}" failed. No retries permitted until 2026-07-29 20:25:10.159850046 +0000 UTC m=+37.203542207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls") pod "dns-default-cgmsd" (UID: "c106d721-0381-4f51-ad74-dcb1804221fc") : secret "dns-default-metrics-tls" not found
Jul 29 20:25:09.159882 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:09.159876    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls podName:2883174b-258b-4a7b-906d-237dc51e8027 nodeName:}" failed. No retries permitted until 2026-07-29 20:25:10.159869654 +0000 UTC m=+37.203561815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls") pod "image-registry-6bf8fdf9cc-85tqw" (UID: "2883174b-258b-4a7b-906d-237dc51e8027") : secret "image-registry-tls" not found
Jul 29 20:25:09.518218 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:09.518181    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:09.521455 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:09.521434    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 20:25:09.807066 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:09.806988    2585 generic.go:356] "Generic (PLEG): container finished" podID="a78b2c7b-217a-4490-921d-84357434b580" containerID="0d37ddbfdf10794a921160e8182bb79124d819b4bec286773938d2ca7e606041" exitCode=0
Jul 29 20:25:09.807216 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:09.807072    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7z62g" event={"ID":"a78b2c7b-217a-4490-921d-84357434b580","Type":"ContainerDied","Data":"0d37ddbfdf10794a921160e8182bb79124d819b4bec286773938d2ca7e606041"}
Jul 29 20:25:09.808146 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:09.808127    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-knr4r" event={"ID":"48bb1c6c-0d0e-436c-af8d-747647703600","Type":"ContainerStarted","Data":"f693653d02c8675d7419ab255c4196b20fa52748db3f8d344d9ac348948f217b"}
Jul 29 20:25:10.165437 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:10.165359    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:10.165663 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:10.165450    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:10.165663 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:10.165511    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:25:10.165663 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:10.165537    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf8fdf9cc-85tqw: secret "image-registry-tls" not found
Jul 29 20:25:10.165663 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:10.165573    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:25:10.165663 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:10.165608    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls podName:2883174b-258b-4a7b-906d-237dc51e8027 nodeName:}" failed. No retries permitted until 2026-07-29 20:25:12.16558766 +0000 UTC m=+39.209279843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls") pod "image-registry-6bf8fdf9cc-85tqw" (UID: "2883174b-258b-4a7b-906d-237dc51e8027") : secret "image-registry-tls" not found
Jul 29 20:25:10.165663 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:10.165626    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls podName:c106d721-0381-4f51-ad74-dcb1804221fc nodeName:}" failed. No retries permitted until 2026-07-29 20:25:12.165617542 +0000 UTC m=+39.209309703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls") pod "dns-default-cgmsd" (UID: "c106d721-0381-4f51-ad74-dcb1804221fc") : secret "dns-default-metrics-tls" not found
Jul 29 20:25:10.813779 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:10.813743    2585 generic.go:356] "Generic (PLEG): container finished" podID="a78b2c7b-217a-4490-921d-84357434b580" containerID="09cdf5d193c5db8cb689e828d433d8eb629403d5c448e84b9e49b89a3ebe63e3" exitCode=0
Jul 29 20:25:10.814236 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:10.813826    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7z62g" event={"ID":"a78b2c7b-217a-4490-921d-84357434b580","Type":"ContainerDied","Data":"09cdf5d193c5db8cb689e828d433d8eb629403d5c448e84b9e49b89a3ebe63e3"}
Jul 29 20:25:11.819218 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:11.819047    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7z62g" event={"ID":"a78b2c7b-217a-4490-921d-84357434b580","Type":"ContainerStarted","Data":"f7cb3de5f372fbb57318857181c1f55ad3e3a0b889c0df8d1f9e85061ec0db82"}
Jul 29 20:25:11.849300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:11.849262    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7z62g" podStartSLOduration=8.578693034 podStartE2EDuration="37.84924642s" totalImagesPullingTime="29.270553386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:25:11.847788472 +0000 UTC m=+38.891480665" watchObservedRunningTime="2026-07-29 20:25:11.84924642 +0000 UTC m=+38.892938597"
Jul 29 20:25:11.967440 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:11.966128    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:12.065094 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:12.065063    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:12.179901 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:12.179824    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:12.179901 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:12.179885    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:12.180061 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:12.179971    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:25:12.180061 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:12.179988    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf8fdf9cc-85tqw: secret "image-registry-tls" not found
Jul 29 20:25:12.180061 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:12.179990    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:25:12.180061 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:12.180040    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls podName:c106d721-0381-4f51-ad74-dcb1804221fc nodeName:}" failed. No retries permitted until 2026-07-29 20:25:16.180022365 +0000 UTC m=+43.223714527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls") pod "dns-default-cgmsd" (UID: "c106d721-0381-4f51-ad74-dcb1804221fc") : secret "dns-default-metrics-tls" not found
Jul 29 20:25:12.180061 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:12.180060    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls podName:2883174b-258b-4a7b-906d-237dc51e8027 nodeName:}" failed. No retries permitted until 2026-07-29 20:25:16.180049793 +0000 UTC m=+43.223741968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls") pod "image-registry-6bf8fdf9cc-85tqw" (UID: "2883174b-258b-4a7b-906d-237dc51e8027") : secret "image-registry-tls" not found
Jul 29 20:25:12.184039 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:12.184013    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:12.268897 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:12.268868    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:12.822383 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:12.822350    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-knr4r" event={"ID":"48bb1c6c-0d0e-436c-af8d-747647703600","Type":"ContainerStarted","Data":"c4940146b55684dcc04ee6ecf003b4d520738562a2059a9e223337acf42d3e66"}
Jul 29 20:25:12.844740 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:12.844693    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-knr4r" podStartSLOduration=1.886592823 podStartE2EDuration="4.844677089s" totalImagesPullingTime="2.958084266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:25:12.843046441 +0000 UTC m=+39.886738624" watchObservedRunningTime="2026-07-29 20:25:12.844677089 +0000 UTC m=+39.888369271"
Jul 29 20:25:13.049460 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:13.049414    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:13.150100 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:13.150032    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:13.309550 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:13.309503    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:13.380579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:13.380547    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:14.135322 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:14.135280    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:14.225018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:14.224984    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:16.202173 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:16.202137    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:16.202530 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:16.202187    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:16.202530 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:16.202276    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:25:16.202530 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:16.202349    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls podName:c106d721-0381-4f51-ad74-dcb1804221fc nodeName:}" failed. No retries permitted until 2026-07-29 20:25:24.202316184 +0000 UTC m=+51.246008345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls") pod "dns-default-cgmsd" (UID: "c106d721-0381-4f51-ad74-dcb1804221fc") : secret "dns-default-metrics-tls" not found
Jul 29 20:25:16.202530 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:16.202284    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:25:16.202530 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:16.202370    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf8fdf9cc-85tqw: secret "image-registry-tls" not found
Jul 29 20:25:16.202530 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:16.202419    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls podName:2883174b-258b-4a7b-906d-237dc51e8027 nodeName:}" failed. No retries permitted until 2026-07-29 20:25:24.202405949 +0000 UTC m=+51.246098113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls") pod "image-registry-6bf8fdf9cc-85tqw" (UID: "2883174b-258b-4a7b-906d-237dc51e8027") : secret "image-registry-tls" not found
Jul 29 20:25:18.448822 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:18.448795    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:24.256848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:24.256809    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:24.257294 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:24.256861    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:24.257294 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:24.256959    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:25:24.257294 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:24.256963    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:25:24.257294 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:24.257030    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls podName:c106d721-0381-4f51-ad74-dcb1804221fc nodeName:}" failed. No retries permitted until 2026-07-29 20:25:40.257011847 +0000 UTC m=+67.300704008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls") pod "dns-default-cgmsd" (UID: "c106d721-0381-4f51-ad74-dcb1804221fc") : secret "dns-default-metrics-tls" not found
Jul 29 20:25:24.257294 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:24.256970    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf8fdf9cc-85tqw: secret "image-registry-tls" not found
Jul 29 20:25:24.257294 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:24.257093    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls podName:2883174b-258b-4a7b-906d-237dc51e8027 nodeName:}" failed. No retries permitted until 2026-07-29 20:25:40.257081749 +0000 UTC m=+67.300773910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls") pod "image-registry-6bf8fdf9cc-85tqw" (UID: "2883174b-258b-4a7b-906d-237dc51e8027") : secret "image-registry-tls" not found
Jul 29 20:25:25.139317 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:25.139282    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:27.024523 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:27.024491    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:27.156689 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:27.156661    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:27.299613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:27.299539    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:27.398802 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:27.398743    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:27.498919 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:27.498887    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:28.405855 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:28.405823    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:28.478793 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:28.478764    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:29.336710 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:29.336678    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:29.437772 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:29.437742    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:30.394452 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:30.394396    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:32.793260 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:32.793233    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lprd"
Jul 29 20:25:32.973788 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:32.973754    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:33.076044 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:33.075976    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:39.073090 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.073047    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:39.162494 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.162460    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:25:39.165216 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.165196    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 20:25:39.173471 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:39.173452    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 20:25:39.173579 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:39.173543    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs podName:94bf5594-165a-4efb-ba8c-a628ea34eabf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:43.173514462 +0000 UTC m=+130.217206639 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs") pod "network-metrics-daemon-7sqwj" (UID: "94bf5594-165a-4efb-ba8c-a628ea34eabf") : secret "metrics-daemon-secret" not found
Jul 29 20:25:39.263470 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.263432    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:39.263651 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.263540    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:39.266518 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.266495    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 20:25:39.266583 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.266552    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 20:25:39.276499 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.276466    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 20:25:39.277708 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.277689    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0c1248c-5bb1-4cd3-bade-adc29339979c-original-pull-secret\") pod \"global-pull-secret-syncer-9xh8g\" (UID: \"b0c1248c-5bb1-4cd3-bade-adc29339979c\") " pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:39.286886 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.286861    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngk9r\" (UniqueName: \"kubernetes.io/projected/a0d385ca-876f-4748-a57c-8e0c3f5ca7cc-kube-api-access-ngk9r\") pod \"network-check-target-zmnd7\" (UID: \"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc\") " pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:39.435359 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.435280    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mkp42\""
Jul 29 20:25:39.442987 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.442969    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:39.528715 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.528687    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xh8g"
Jul 29 20:25:39.587122 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.587097    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zmnd7"]
Jul 29 20:25:39.592356 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:25:39.592311    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d385ca_876f_4748_a57c_8e0c3f5ca7cc.slice/crio-553e9634a24d67e97b6ac5d602506518cca86de2a7defe463c582083e65acebe WatchSource:0}: Error finding container 553e9634a24d67e97b6ac5d602506518cca86de2a7defe463c582083e65acebe: Status 404 returned error can't find the container with id 553e9634a24d67e97b6ac5d602506518cca86de2a7defe463c582083e65acebe
Jul 29 20:25:39.651679 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.651651    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9xh8g"]
Jul 29 20:25:39.654280 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:25:39.654254    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c1248c_5bb1_4cd3_bade_adc29339979c.slice/crio-ece7ce850ba7fee4d95d1c6205d52ab9b8b42f9415817a569ab72ca82e028bfb WatchSource:0}: Error finding container ece7ce850ba7fee4d95d1c6205d52ab9b8b42f9415817a569ab72ca82e028bfb: Status 404 returned error can't find the container with id ece7ce850ba7fee4d95d1c6205d52ab9b8b42f9415817a569ab72ca82e028bfb
Jul 29 20:25:39.884413 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.884373    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9xh8g" event={"ID":"b0c1248c-5bb1-4cd3-bade-adc29339979c","Type":"ContainerStarted","Data":"ece7ce850ba7fee4d95d1c6205d52ab9b8b42f9415817a569ab72ca82e028bfb"}
Jul 29 20:25:39.885501 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:39.885477    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zmnd7" event={"ID":"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc","Type":"ContainerStarted","Data":"553e9634a24d67e97b6ac5d602506518cca86de2a7defe463c582083e65acebe"}
Jul 29 20:25:40.270322 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:40.270290    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:25:40.270859 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:40.270379    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:25:40.270859 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:40.270463    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:25:40.270859 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:40.270489    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:25:40.270859 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:40.270504    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bf8fdf9cc-85tqw: secret "image-registry-tls" not found
Jul 29 20:25:40.270859 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:40.270551    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls podName:c106d721-0381-4f51-ad74-dcb1804221fc nodeName:}" failed. No retries permitted until 2026-07-29 20:26:12.270527466 +0000 UTC m=+99.314219627 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls") pod "dns-default-cgmsd" (UID: "c106d721-0381-4f51-ad74-dcb1804221fc") : secret "dns-default-metrics-tls" not found
Jul 29 20:25:40.270859 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:40.270574    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls podName:2883174b-258b-4a7b-906d-237dc51e8027 nodeName:}" failed. No retries permitted until 2026-07-29 20:26:12.270562835 +0000 UTC m=+99.314255003 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls") pod "image-registry-6bf8fdf9cc-85tqw" (UID: "2883174b-258b-4a7b-906d-237dc51e8027") : secret "image-registry-tls" not found
Jul 29 20:25:43.437595 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:43.437562    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:43.499755 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:43.499716    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:44.899229 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:44.899194    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9xh8g" event={"ID":"b0c1248c-5bb1-4cd3-bade-adc29339979c","Type":"ContainerStarted","Data":"bc4d4b14a028a9efba6198202cf06273f99df71830a9b7385e1161f04fc3af24"}
Jul 29 20:25:44.900581 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:44.900555    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zmnd7" event={"ID":"a0d385ca-876f-4748-a57c-8e0c3f5ca7cc","Type":"ContainerStarted","Data":"66a55d05d4a55fcbd7fc71e928241ad2000b87d2e91243c068e9192813bfd1a6"}
Jul 29 20:25:44.900691 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:44.900679    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:25:44.914082 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:44.914036    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9xh8g" podStartSLOduration=66.282079015 podStartE2EDuration="1m10.914021081s" totalImagesPullingTime="4.631942066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:25:44.913668292 +0000 UTC m=+71.957360477" watchObservedRunningTime="2026-07-29 20:25:44.914021081 +0000 UTC m=+71.957713265"
Jul 29 20:25:44.928602 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:44.928559    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zmnd7" podStartSLOduration=66.245765662 podStartE2EDuration="1m10.928549413s" totalImagesPullingTime="4.682783751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:25:44.928523021 +0000 UTC m=+71.972215204" watchObservedRunningTime="2026-07-29 20:25:44.928549413 +0000 UTC m=+71.972241595"
Jul 29 20:25:50.010267 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:50.010226    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:50.070563 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:50.070534    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:55.722847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.722815    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"]
Jul 29 20:25:55.724847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.724828    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.727615 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.727588    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 29 20:25:55.727720 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.727600    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 29 20:25:55.728890 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.728874    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:25:55.728965 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.728895    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4m8hx\""
Jul 29 20:25:55.729019 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.728970    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 29 20:25:55.737008 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.736991    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"]
Jul 29 20:25:55.770430 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.770402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7bc6de2-919c-4f33-98ea-1a838956bd8f-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl\" (UID: \"e7bc6de2-919c-4f33-98ea-1a838956bd8f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.770539 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.770453    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7bc6de2-919c-4f33-98ea-1a838956bd8f-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl\" (UID: \"e7bc6de2-919c-4f33-98ea-1a838956bd8f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.770589 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.770533    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt442\" (UniqueName: \"kubernetes.io/projected/e7bc6de2-919c-4f33-98ea-1a838956bd8f-kube-api-access-tt442\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl\" (UID: \"e7bc6de2-919c-4f33-98ea-1a838956bd8f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.811006 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.810982    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"]
Jul 29 20:25:55.812733 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.812719    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:25:55.816278 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.816260    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 29 20:25:55.816789 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.816766    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:25:55.816870 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.816789    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r8xl4\""
Jul 29 20:25:55.816870 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.816827    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 29 20:25:55.830723 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.830704    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"]
Jul 29 20:25:55.871163 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.871140    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt442\" (UniqueName: \"kubernetes.io/projected/e7bc6de2-919c-4f33-98ea-1a838956bd8f-kube-api-access-tt442\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl\" (UID: \"e7bc6de2-919c-4f33-98ea-1a838956bd8f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.871277 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.871172    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:25:55.871277 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.871209    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7bc6de2-919c-4f33-98ea-1a838956bd8f-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl\" (UID: \"e7bc6de2-919c-4f33-98ea-1a838956bd8f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.871277 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.871235    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7bc6de2-919c-4f33-98ea-1a838956bd8f-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl\" (UID: \"e7bc6de2-919c-4f33-98ea-1a838956bd8f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.871427 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.871276    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfxb\" (UniqueName: \"kubernetes.io/projected/9bf5fa35-de9b-4d20-811a-d8f4a4894544-kube-api-access-wmfxb\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:25:55.871693 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.871667    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7bc6de2-919c-4f33-98ea-1a838956bd8f-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl\" (UID: \"e7bc6de2-919c-4f33-98ea-1a838956bd8f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.873301 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.873285    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7bc6de2-919c-4f33-98ea-1a838956bd8f-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl\" (UID: \"e7bc6de2-919c-4f33-98ea-1a838956bd8f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.886966 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.886940    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt442\" (UniqueName: \"kubernetes.io/projected/e7bc6de2-919c-4f33-98ea-1a838956bd8f-kube-api-access-tt442\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl\" (UID: \"e7bc6de2-919c-4f33-98ea-1a838956bd8f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:55.907896 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.907869    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-7lq85"]
Jul 29 20:25:55.909736 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.909718    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:55.913466 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.913448    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 29 20:25:55.913669 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.913653    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 29 20:25:55.913736 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.913681    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xcbsz\""
Jul 29 20:25:55.913736 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.913661    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:25:55.914060 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.914043    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 29 20:25:55.919524 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.919505    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 29 20:25:55.924895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.924875    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-7lq85"]
Jul 29 20:25:55.971627 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.971600    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0248d8c6-87bb-4df2-8f61-f80a2c51469b-config\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:55.971744 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.971633    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:25:55.971744 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.971710    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0248d8c6-87bb-4df2-8f61-f80a2c51469b-trusted-ca\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:55.971818 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:55.971744    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 20:25:55.971818 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.971743    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0248d8c6-87bb-4df2-8f61-f80a2c51469b-serving-cert\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:55.971818 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:55.971795    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls podName:9bf5fa35-de9b-4d20-811a-d8f4a4894544 nodeName:}" failed. No retries permitted until 2026-07-29 20:25:56.471782839 +0000 UTC m=+83.515475000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls") pod "cluster-samples-operator-6748489656-4hfnx" (UID: "9bf5fa35-de9b-4d20-811a-d8f4a4894544") : secret "samples-operator-tls" not found
Jul 29 20:25:55.971911 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.971835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfxb\" (UniqueName: \"kubernetes.io/projected/9bf5fa35-de9b-4d20-811a-d8f4a4894544-kube-api-access-wmfxb\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:25:55.971911 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.971862    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shw24\" (UniqueName: \"kubernetes.io/projected/0248d8c6-87bb-4df2-8f61-f80a2c51469b-kube-api-access-shw24\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:55.980942 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:55.980888    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfxb\" (UniqueName: \"kubernetes.io/projected/9bf5fa35-de9b-4d20-811a-d8f4a4894544-kube-api-access-wmfxb\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:25:56.008311 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.008280    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:56.033222 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.033198    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"
Jul 29 20:25:56.072822 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.072783    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shw24\" (UniqueName: \"kubernetes.io/projected/0248d8c6-87bb-4df2-8f61-f80a2c51469b-kube-api-access-shw24\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:56.072970 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.072859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0248d8c6-87bb-4df2-8f61-f80a2c51469b-config\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:56.072970 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.072948    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0248d8c6-87bb-4df2-8f61-f80a2c51469b-trusted-ca\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:56.073086 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.072979    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0248d8c6-87bb-4df2-8f61-f80a2c51469b-serving-cert\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:56.073593 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.073550    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0248d8c6-87bb-4df2-8f61-f80a2c51469b-config\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:56.073863 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.073842    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0248d8c6-87bb-4df2-8f61-f80a2c51469b-trusted-ca\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:56.075219 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.075194    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0248d8c6-87bb-4df2-8f61-f80a2c51469b-serving-cert\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:56.082778 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.082752    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shw24\" (UniqueName: \"kubernetes.io/projected/0248d8c6-87bb-4df2-8f61-f80a2c51469b-kube-api-access-shw24\") pod \"console-operator-584dcb6b54-7lq85\" (UID: \"0248d8c6-87bb-4df2-8f61-f80a2c51469b\") " pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:56.108196 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.108147    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:56.151719 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.151692    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl"]
Jul 29 20:25:56.154497 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:25:56.154473    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bc6de2_919c_4f33_98ea_1a838956bd8f.slice/crio-bf7e2a4b1751fbcdb24a1f13b95a3a777d388759fe06ec051ad36ac2818ba8bf WatchSource:0}: Error finding container bf7e2a4b1751fbcdb24a1f13b95a3a777d388759fe06ec051ad36ac2818ba8bf: Status 404 returned error can't find the container with id bf7e2a4b1751fbcdb24a1f13b95a3a777d388759fe06ec051ad36ac2818ba8bf
Jul 29 20:25:56.218542 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.218520    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:25:56.334036 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.334002    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-7lq85"]
Jul 29 20:25:56.337236 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:25:56.337211    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0248d8c6_87bb_4df2_8f61_f80a2c51469b.slice/crio-5501b832e096d46fc05a2efb081d4caaa9018e65abdf302c5ebfce7dbebe472f WatchSource:0}: Error finding container 5501b832e096d46fc05a2efb081d4caaa9018e65abdf302c5ebfce7dbebe472f: Status 404 returned error can't find the container with id 5501b832e096d46fc05a2efb081d4caaa9018e65abdf302c5ebfce7dbebe472f
Jul 29 20:25:56.475657 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.475622    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:25:56.475778 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:56.475760    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 20:25:56.475832 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:56.475823    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls podName:9bf5fa35-de9b-4d20-811a-d8f4a4894544 nodeName:}" failed. No retries permitted until 2026-07-29 20:25:57.475808427 +0000 UTC m=+84.519500591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls") pod "cluster-samples-operator-6748489656-4hfnx" (UID: "9bf5fa35-de9b-4d20-811a-d8f4a4894544") : secret "samples-operator-tls" not found
Jul 29 20:25:56.929245 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.929207    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl" event={"ID":"e7bc6de2-919c-4f33-98ea-1a838956bd8f","Type":"ContainerStarted","Data":"bf7e2a4b1751fbcdb24a1f13b95a3a777d388759fe06ec051ad36ac2818ba8bf"}
Jul 29 20:25:56.930211 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:56.930189    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" event={"ID":"0248d8c6-87bb-4df2-8f61-f80a2c51469b","Type":"ContainerStarted","Data":"5501b832e096d46fc05a2efb081d4caaa9018e65abdf302c5ebfce7dbebe472f"}
Jul 29 20:25:57.483834 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:57.483757    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:25:57.484014 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:57.483893    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 20:25:57.484014 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:57.483966    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls podName:9bf5fa35-de9b-4d20-811a-d8f4a4894544 nodeName:}" failed. No retries permitted until 2026-07-29 20:25:59.483946487 +0000 UTC m=+86.527638647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls") pod "cluster-samples-operator-6748489656-4hfnx" (UID: "9bf5fa35-de9b-4d20-811a-d8f4a4894544") : secret "samples-operator-tls" not found
Jul 29 20:25:59.080056 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:59.080000    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:59.180679 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:59.180648    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:25:59.498898 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:59.498872    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:25:59.499024 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:59.498983    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 20:25:59.499065 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:25:59.499035    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls podName:9bf5fa35-de9b-4d20-811a-d8f4a4894544 nodeName:}" failed. No retries permitted until 2026-07-29 20:26:03.499022409 +0000 UTC m=+90.542714570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls") pod "cluster-samples-operator-6748489656-4hfnx" (UID: "9bf5fa35-de9b-4d20-811a-d8f4a4894544") : secret "samples-operator-tls" not found
Jul 29 20:25:59.939757 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:59.939685    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/0.log"
Jul 29 20:25:59.939757 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:59.939725    2585 generic.go:356] "Generic (PLEG): container finished" podID="0248d8c6-87bb-4df2-8f61-f80a2c51469b" containerID="2c4a6521ff824af3e05598e0d81ec7a94e28c18f426373a9522e52de0fdfced2" exitCode=255
Jul 29 20:25:59.939962 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:59.939823    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" event={"ID":"0248d8c6-87bb-4df2-8f61-f80a2c51469b","Type":"ContainerDied","Data":"2c4a6521ff824af3e05598e0d81ec7a94e28c18f426373a9522e52de0fdfced2"}
Jul 29 20:25:59.940046 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:59.940032    2585 scope.go:117] "RemoveContainer" containerID="2c4a6521ff824af3e05598e0d81ec7a94e28c18f426373a9522e52de0fdfced2"
Jul 29 20:25:59.941425 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:59.941404    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl" event={"ID":"e7bc6de2-919c-4f33-98ea-1a838956bd8f","Type":"ContainerStarted","Data":"4dc33391f3497c18523abd8d9cc218a652f47ac528d35200bc38fa44ff7f1f6a"}
Jul 29 20:25:59.973610 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:25:59.973555    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl" podStartSLOduration=1.959711885 podStartE2EDuration="4.973538407s" totalImagesPullingTime="3.013826522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:25:59.973089454 +0000 UTC m=+87.016781637" watchObservedRunningTime="2026-07-29 20:25:59.973538407 +0000 UTC m=+87.017230591"
Jul 29 20:26:00.438727 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.438695    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ggdgh"]
Jul 29 20:26:00.440802 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.440787    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.443756 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.443738    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 20:26:00.446030 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.446010    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 20:26:00.446360 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.446344    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 20:26:00.446452 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.446432    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 20:26:00.447502 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.447480    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bvgmt\""
Jul 29 20:26:00.459948 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.459928    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ggdgh"]
Jul 29 20:26:00.506089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.506064    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.506174 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.506105    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-crio-socket\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.506174 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.506158    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.506243 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.506229    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4nr7\" (UniqueName: \"kubernetes.io/projected/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-api-access-f4nr7\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.506294 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.506282    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-data-volume\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.526421 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.526389    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:00.606724 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.606699    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.606807 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.606767    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4nr7\" (UniqueName: \"kubernetes.io/projected/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-api-access-f4nr7\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.606807 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:00.606775    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:26:00.606920 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.606822    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-data-volume\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.606920 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:00.606835    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:01.106822208 +0000 UTC m=+88.150514389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : configmap "kube-rbac-proxy" not found
Jul 29 20:26:00.606920 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.606871    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.606920 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.606901    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-crio-socket\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.607130 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.606964    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-crio-socket\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.607130 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:00.607004    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:00.607130 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:00.607046    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:01.107035783 +0000 UTC m=+88.150727971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:00.607130 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.607109    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-data-volume\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.616388 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.616358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4nr7\" (UniqueName: \"kubernetes.io/projected/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-api-access-f4nr7\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:00.733787 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.733759    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:00.939521 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.939492    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:00.946536 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.946512    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/1.log"
Jul 29 20:26:00.946939 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.946925    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/0.log"
Jul 29 20:26:00.947018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.946958    2585 generic.go:356] "Generic (PLEG): container finished" podID="0248d8c6-87bb-4df2-8f61-f80a2c51469b" containerID="73664de1dab7d5d2a4f8ba333d4a58de3a935dce79b2d2243bcfeb7d29a419ef" exitCode=255
Jul 29 20:26:00.947018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.946987    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" event={"ID":"0248d8c6-87bb-4df2-8f61-f80a2c51469b","Type":"ContainerDied","Data":"73664de1dab7d5d2a4f8ba333d4a58de3a935dce79b2d2243bcfeb7d29a419ef"}
Jul 29 20:26:00.947123 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.947039    2585 scope.go:117] "RemoveContainer" containerID="2c4a6521ff824af3e05598e0d81ec7a94e28c18f426373a9522e52de0fdfced2"
Jul 29 20:26:00.947312 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:00.947278    2585 scope.go:117] "RemoveContainer" containerID="73664de1dab7d5d2a4f8ba333d4a58de3a935dce79b2d2243bcfeb7d29a419ef"
Jul 29 20:26:00.947487 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:00.947468    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-7lq85_openshift-console-operator(0248d8c6-87bb-4df2-8f61-f80a2c51469b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" podUID="0248d8c6-87bb-4df2-8f61-f80a2c51469b"
Jul 29 20:26:01.012053 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:01.011989    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:01.111819 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:01.111795    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:01.111937 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:01.111857    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:01.111937 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:01.111928    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:01.112033 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:01.111980    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:26:01.112033 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:01.111996    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:02.111981203 +0000 UTC m=+89.155673369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:01.112033 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:01.112033    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:02.112020079 +0000 UTC m=+89.155712243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : configmap "kube-rbac-proxy" not found
Jul 29 20:26:01.206446 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:01.206419    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:01.293350 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:01.293266    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:01.936111 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:01.936078    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:01.951219 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:01.951200    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/1.log"
Jul 29 20:26:01.951520 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:01.951503    2585 scope.go:117] "RemoveContainer" containerID="73664de1dab7d5d2a4f8ba333d4a58de3a935dce79b2d2243bcfeb7d29a419ef"
Jul 29 20:26:01.951662 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:01.951646    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-7lq85_openshift-console-operator(0248d8c6-87bb-4df2-8f61-f80a2c51469b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" podUID="0248d8c6-87bb-4df2-8f61-f80a2c51469b"
Jul 29 20:26:02.119721 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.119694    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:02.119840 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.119793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:02.119840 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:02.119799    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:26:02.119952 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:02.119844    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:02.119952 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:02.119887    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:04.119868184 +0000 UTC m=+91.163560366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : configmap "kube-rbac-proxy" not found
Jul 29 20:26:02.119952 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:02.119906    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:04.119897006 +0000 UTC m=+91.163589171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:02.231092 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.231072    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:02.332126 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.332083    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:02.631156 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.631086    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qm8fd_9191e46f-8231-4d1c-8104-c679ad0e2eeb/dns-node-resolver/0.log"
Jul 29 20:26:02.806310 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.806284    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-dbkjq"]
Jul 29 20:26:02.810413 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.810393    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:02.813323 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.813302    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 20:26:02.814547 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.814521    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 20:26:02.814547 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.814535    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ksxwm\""
Jul 29 20:26:02.814656 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.814521    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 20:26:02.814656 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.814551    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 20:26:02.819098 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.819078    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-dbkjq"]
Jul 29 20:26:02.926611 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.926554    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9306a461-0a42-42ec-90f9-57da6ab22b7a-signing-key\") pod \"service-ca-658d7b7c78-dbkjq\" (UID: \"9306a461-0a42-42ec-90f9-57da6ab22b7a\") " pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:02.926611 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.926594    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9306a461-0a42-42ec-90f9-57da6ab22b7a-signing-cabundle\") pod \"service-ca-658d7b7c78-dbkjq\" (UID: \"9306a461-0a42-42ec-90f9-57da6ab22b7a\") " pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:02.926732 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.926648    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clgml\" (UniqueName: \"kubernetes.io/projected/9306a461-0a42-42ec-90f9-57da6ab22b7a-kube-api-access-clgml\") pod \"service-ca-658d7b7c78-dbkjq\" (UID: \"9306a461-0a42-42ec-90f9-57da6ab22b7a\") " pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:02.996418 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:02.996394    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:03.027614 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.027590    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clgml\" (UniqueName: \"kubernetes.io/projected/9306a461-0a42-42ec-90f9-57da6ab22b7a-kube-api-access-clgml\") pod \"service-ca-658d7b7c78-dbkjq\" (UID: \"9306a461-0a42-42ec-90f9-57da6ab22b7a\") " pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:03.027681 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.027654    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9306a461-0a42-42ec-90f9-57da6ab22b7a-signing-key\") pod \"service-ca-658d7b7c78-dbkjq\" (UID: \"9306a461-0a42-42ec-90f9-57da6ab22b7a\") " pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:03.027723 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.027693    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9306a461-0a42-42ec-90f9-57da6ab22b7a-signing-cabundle\") pod \"service-ca-658d7b7c78-dbkjq\" (UID: \"9306a461-0a42-42ec-90f9-57da6ab22b7a\") " pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:03.028396 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.028377    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9306a461-0a42-42ec-90f9-57da6ab22b7a-signing-cabundle\") pod \"service-ca-658d7b7c78-dbkjq\" (UID: \"9306a461-0a42-42ec-90f9-57da6ab22b7a\") " pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:03.030040 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.030020    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9306a461-0a42-42ec-90f9-57da6ab22b7a-signing-key\") pod \"service-ca-658d7b7c78-dbkjq\" (UID: \"9306a461-0a42-42ec-90f9-57da6ab22b7a\") " pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:03.036084 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.036066    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clgml\" (UniqueName: \"kubernetes.io/projected/9306a461-0a42-42ec-90f9-57da6ab22b7a-kube-api-access-clgml\") pod \"service-ca-658d7b7c78-dbkjq\" (UID: \"9306a461-0a42-42ec-90f9-57da6ab22b7a\") " pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:03.061906 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.061885    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:03.120440 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.120404    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq"
Jul 29 20:26:03.231489 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.231467    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k58wb_7cf47463-470a-4c86-8301-a838734a4963/node-ca/0.log"
Jul 29 20:26:03.237674 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.237583    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-dbkjq"]
Jul 29 20:26:03.240144 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:03.240120    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9306a461_0a42_42ec_90f9_57da6ab22b7a.slice/crio-bb412100ab393686962fc56c0e79e8b7af2a7c57700a7b50034186332e13f1eb WatchSource:0}: Error finding container bb412100ab393686962fc56c0e79e8b7af2a7c57700a7b50034186332e13f1eb: Status 404 returned error can't find the container with id bb412100ab393686962fc56c0e79e8b7af2a7c57700a7b50034186332e13f1eb
Jul 29 20:26:03.530939 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.530912    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:26:03.531110 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:03.531034    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 20:26:03.531110 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:03.531099    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls podName:9bf5fa35-de9b-4d20-811a-d8f4a4894544 nodeName:}" failed. No retries permitted until 2026-07-29 20:26:11.531086255 +0000 UTC m=+98.574778428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls") pod "cluster-samples-operator-6748489656-4hfnx" (UID: "9bf5fa35-de9b-4d20-811a-d8f4a4894544") : secret "samples-operator-tls" not found
Jul 29 20:26:03.800136 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.800049    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:03.958957 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:03.958901    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq" event={"ID":"9306a461-0a42-42ec-90f9-57da6ab22b7a","Type":"ContainerStarted","Data":"bb412100ab393686962fc56c0e79e8b7af2a7c57700a7b50034186332e13f1eb"}
Jul 29 20:26:04.135896 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:04.135806    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:04.136312 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:04.135934    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:04.136312 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:04.135954    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:26:04.136312 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:04.136035    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:08.136016026 +0000 UTC m=+95.179708192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : configmap "kube-rbac-proxy" not found
Jul 29 20:26:04.136312 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:04.136057    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:04.136312 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:04.136099    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:08.136088969 +0000 UTC m=+95.179781133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:04.231154 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:04.231126    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-knr4r_48bb1c6c-0d0e-436c-af8d-747647703600/serve-healthcheck-canary/0.log"
Jul 29 20:26:05.965580 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:05.965535    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq" event={"ID":"9306a461-0a42-42ec-90f9-57da6ab22b7a","Type":"ContainerStarted","Data":"3e22bb7e12599936de346eb3f5aae5f562841bfd446fb3d239ba586aa2e3652e"}
Jul 29 20:26:05.982874 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:05.982825    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-dbkjq" podStartSLOduration=2.219376548 podStartE2EDuration="3.982809341s" totalImagesPullingTime="1.763432793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:05.981731277 +0000 UTC m=+93.025423460" watchObservedRunningTime="2026-07-29 20:26:05.982809341 +0000 UTC m=+93.026501548"
Jul 29 20:26:06.219209 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:06.219126    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:26:06.219209 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:06.219176    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:26:06.219504 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:06.219491    2585 scope.go:117] "RemoveContainer" containerID="73664de1dab7d5d2a4f8ba333d4a58de3a935dce79b2d2243bcfeb7d29a419ef"
Jul 29 20:26:06.219648 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:06.219630    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-7lq85_openshift-console-operator(0248d8c6-87bb-4df2-8f61-f80a2c51469b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" podUID="0248d8c6-87bb-4df2-8f61-f80a2c51469b"
Jul 29 20:26:06.862921 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:06.862886    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:07.004063 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:07.004033    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:07.133910 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:07.133808    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:07.210719 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:07.210686    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:07.345072 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:07.345033    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:07.531565 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:07.531536    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:07.603280 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:07.603254    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:08.150394 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:08.150361    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:08.170243 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:08.170217    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:08.170384 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:08.170271    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:08.170436 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:08.170383    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:08.170436 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:08.170399    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:26:08.170498 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:08.170438    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:16.170421695 +0000 UTC m=+103.214113860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:08.170498 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:08.170451    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:16.170445524 +0000 UTC m=+103.214137685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : configmap "kube-rbac-proxy" not found
Jul 29 20:26:08.716236 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:08.716200    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:09.111867 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:09.111772    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:09.518559 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:09.518527    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:09.719645 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:09.719618    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:10.118291 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:10.118247    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:10.518074 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:10.518041    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:10.711536 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:10.711504    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:10.812521 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:10.812435    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:11.118447 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:11.118363    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:11.317419 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:11.317386    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:11.379799 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:11.379726    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:11.595112 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:11.595082    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:26:11.597423 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:11.597391    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9bf5fa35-de9b-4d20-811a-d8f4a4894544-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4hfnx\" (UID: \"9bf5fa35-de9b-4d20-811a-d8f4a4894544\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:26:11.713706 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:11.713647    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:11.721403 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:11.721384    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"
Jul 29 20:26:11.841561 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:11.841534    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx"]
Jul 29 20:26:11.987502 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:11.987474    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx" event={"ID":"9bf5fa35-de9b-4d20-811a-d8f4a4894544","Type":"ContainerStarted","Data":"152f24fc68fe1e115e978b310b0efded1c1c6c1bd43091eb71a6cbe8516daf9c"}
Jul 29 20:26:12.118503 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.118465    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:12.302303 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.302246    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:26:12.302303 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.302295    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:26:12.304526 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.304495    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"image-registry-6bf8fdf9cc-85tqw\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") " pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:26:12.304526 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.304518    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c106d721-0381-4f51-ad74-dcb1804221fc-metrics-tls\") pod \"dns-default-cgmsd\" (UID: \"c106d721-0381-4f51-ad74-dcb1804221fc\") " pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:26:12.569890 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.569818    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-brdv4\""
Jul 29 20:26:12.577079 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.577048    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:26:12.582036 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.582015    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xpqnl\""
Jul 29 20:26:12.589483 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.589459    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:26:12.725397 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.725310    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:12.725916 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.725585    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"]
Jul 29 20:26:12.730543 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:12.730515    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2883174b_258b_4a7b_906d_237dc51e8027.slice/crio-8a9429f1ade29edfbba21097d4f5ae9e23bf44f4017273dca5329deaa1d9f4ec WatchSource:0}: Error finding container 8a9429f1ade29edfbba21097d4f5ae9e23bf44f4017273dca5329deaa1d9f4ec: Status 404 returned error can't find the container with id 8a9429f1ade29edfbba21097d4f5ae9e23bf44f4017273dca5329deaa1d9f4ec
Jul 29 20:26:12.743162 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.743140    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cgmsd"]
Jul 29 20:26:12.745705 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:12.745677    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc106d721_0381_4f51_ad74_dcb1804221fc.slice/crio-e48e78ae5e1b4698b4abe014d9ad96b50face0c2f536a9c635c54fed84a1b0fb WatchSource:0}: Error finding container e48e78ae5e1b4698b4abe014d9ad96b50face0c2f536a9c635c54fed84a1b0fb: Status 404 returned error can't find the container with id e48e78ae5e1b4698b4abe014d9ad96b50face0c2f536a9c635c54fed84a1b0fb
Jul 29 20:26:12.931469 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.931324    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:12.992385 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.992350    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgmsd" event={"ID":"c106d721-0381-4f51-ad74-dcb1804221fc","Type":"ContainerStarted","Data":"e48e78ae5e1b4698b4abe014d9ad96b50face0c2f536a9c635c54fed84a1b0fb"}
Jul 29 20:26:12.994283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.994246    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw" event={"ID":"2883174b-258b-4a7b-906d-237dc51e8027","Type":"ContainerStarted","Data":"47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266"}
Jul 29 20:26:12.994451 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.994293    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw" event={"ID":"2883174b-258b-4a7b-906d-237dc51e8027","Type":"ContainerStarted","Data":"8a9429f1ade29edfbba21097d4f5ae9e23bf44f4017273dca5329deaa1d9f4ec"}
Jul 29 20:26:12.994451 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:12.994422    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:26:13.327087 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:13.327046    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:13.516471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:13.516433    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:13.701884 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:13.701795    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:13.999644 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:13.999610    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx" event={"ID":"9bf5fa35-de9b-4d20-811a-d8f4a4894544","Type":"ContainerStarted","Data":"f1abb5303ab7b046f253a1f457c3a06546a3fa934c96e0a3b2a8bc6be8d76afb"}
Jul 29 20:26:14.083819 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:14.083777    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:14.185433 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:14.185357    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:14.320416 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:14.320323    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:14.715818 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:14.715789    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:14.909020 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:14.908934    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:15.004920 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.004884    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx" event={"ID":"9bf5fa35-de9b-4d20-811a-d8f4a4894544","Type":"ContainerStarted","Data":"808bd2628e4d5d33541c844343d42579c6c2929c654ff1f36b4f73d9d94e6e27"}
Jul 29 20:26:15.006656 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.006634    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgmsd" event={"ID":"c106d721-0381-4f51-ad74-dcb1804221fc","Type":"ContainerStarted","Data":"1e927c87f7a95e85541d4c4d9906de5932f7934609a5bec811b98fa5c53356cf"}
Jul 29 20:26:15.006760 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.006671    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgmsd" event={"ID":"c106d721-0381-4f51-ad74-dcb1804221fc","Type":"ContainerStarted","Data":"2e3c38d5aa8547cfa4a402dde5455b2e2a735946bcf8713c8f98a9dd25b085f4"}
Jul 29 20:26:15.006811 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.006785    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:26:15.023952 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.023900    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4hfnx" podStartSLOduration=18.024264583 podStartE2EDuration="20.023886871s" totalImagesPullingTime="1.999622288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:15.022872125 +0000 UTC m=+102.066564308" watchObservedRunningTime="2026-07-29 20:26:15.023886871 +0000 UTC m=+102.067579054"
Jul 29 20:26:15.024267 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.024243    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw" podStartSLOduration=101.024234679 podStartE2EDuration="1m41.024234679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:13.017013809 +0000 UTC m=+100.060705994" watchObservedRunningTime="2026-07-29 20:26:15.024234679 +0000 UTC m=+102.067926840"
Jul 29 20:26:15.041791 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.041751    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cgmsd" podStartSLOduration=65.317044295 podStartE2EDuration="1m7.041736725s" totalImagesPullingTime="1.72469243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:15.040667232 +0000 UTC m=+102.084359412" watchObservedRunningTime="2026-07-29 20:26:15.041736725 +0000 UTC m=+102.085428908"
Jul 29 20:26:15.340278 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.340240    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:15.529619 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.529589    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:15.905550 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:15.905524    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zmnd7"
Jul 29 20:26:16.120755 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:16.120692    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:16.234108 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:16.234086    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:16.234221 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:16.234140    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:16.234262 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:16.234221    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:16.234297 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:16.234274    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls podName:b8f1b3cb-a4e4-401f-aeb2-34451e025fcf nodeName:}" failed. No retries permitted until 2026-07-29 20:26:32.234258 +0000 UTC m=+119.277950177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggdgh" (UID: "b8f1b3cb-a4e4-401f-aeb2-34451e025fcf") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:26:16.234668 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:16.234645    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:16.318296 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:16.318268    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:16.521007 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:16.520950    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:16.916014 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:16.915948    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:17.314966 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:17.314929    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:17.518255 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:17.518225    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:21.519219 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.519187    2585 scope.go:117] "RemoveContainer" containerID="73664de1dab7d5d2a4f8ba333d4a58de3a935dce79b2d2243bcfeb7d29a419ef"
Jul 29 20:26:21.781973 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.781904    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"]
Jul 29 20:26:21.783883 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.783868    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"
Jul 29 20:26:21.791139 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.788202    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jpx4b\""
Jul 29 20:26:21.791686 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.791646    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 20:26:21.791686 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.791669    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 20:26:21.791879 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.791759    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 20:26:21.791879 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.791760    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 20:26:21.800590 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.800569    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"]
Jul 29 20:26:21.870133 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.870109    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fc36022-29e6-49bd-a054-ab5eb5504e60-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk\" (UID: \"6fc36022-29e6-49bd-a054-ab5eb5504e60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"
Jul 29 20:26:21.870133 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.870135    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkw45\" (UniqueName: \"kubernetes.io/projected/6fc36022-29e6-49bd-a054-ab5eb5504e60-kube-api-access-gkw45\") pod \"managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk\" (UID: \"6fc36022-29e6-49bd-a054-ab5eb5504e60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"
Jul 29 20:26:21.876970 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.876944    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:21.970585 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.970558    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fc36022-29e6-49bd-a054-ab5eb5504e60-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk\" (UID: \"6fc36022-29e6-49bd-a054-ab5eb5504e60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"
Jul 29 20:26:21.970697 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.970587    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkw45\" (UniqueName: \"kubernetes.io/projected/6fc36022-29e6-49bd-a054-ab5eb5504e60-kube-api-access-gkw45\") pod \"managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk\" (UID: \"6fc36022-29e6-49bd-a054-ab5eb5504e60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"
Jul 29 20:26:21.973053 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.973028    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fc36022-29e6-49bd-a054-ab5eb5504e60-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk\" (UID: \"6fc36022-29e6-49bd-a054-ab5eb5504e60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"
Jul 29 20:26:21.973171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.973146    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:21.980606 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:21.980584    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkw45\" (UniqueName: \"kubernetes.io/projected/6fc36022-29e6-49bd-a054-ab5eb5504e60-kube-api-access-gkw45\") pod \"managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk\" (UID: \"6fc36022-29e6-49bd-a054-ab5eb5504e60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"
Jul 29 20:26:22.029711 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:22.029689    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:26:22.030084 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:22.030070    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/1.log"
Jul 29 20:26:22.030161 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:22.030101    2585 generic.go:356] "Generic (PLEG): container finished" podID="0248d8c6-87bb-4df2-8f61-f80a2c51469b" containerID="38191e14b51ebb7857f5bc9892c42fbec99302f5564edc22beeef3d694d957f5" exitCode=255
Jul 29 20:26:22.030161 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:22.030140    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" event={"ID":"0248d8c6-87bb-4df2-8f61-f80a2c51469b","Type":"ContainerDied","Data":"38191e14b51ebb7857f5bc9892c42fbec99302f5564edc22beeef3d694d957f5"}
Jul 29 20:26:22.030255 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:22.030206    2585 scope.go:117] "RemoveContainer" containerID="73664de1dab7d5d2a4f8ba333d4a58de3a935dce79b2d2243bcfeb7d29a419ef"
Jul 29 20:26:22.030547 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:22.030528    2585 scope.go:117] "RemoveContainer" containerID="38191e14b51ebb7857f5bc9892c42fbec99302f5564edc22beeef3d694d957f5"
Jul 29 20:26:22.030740 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:22.030719    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-7lq85_openshift-console-operator(0248d8c6-87bb-4df2-8f61-f80a2c51469b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" podUID="0248d8c6-87bb-4df2-8f61-f80a2c51469b"
Jul 29 20:26:22.060754 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:22.060729    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:22.108510 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:22.108492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"
Jul 29 20:26:22.230256 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:22.230141    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk"]
Jul 29 20:26:22.232611 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:22.232582    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fc36022_29e6_49bd_a054_ab5eb5504e60.slice/crio-984329872cffecc2cd8667b8a40987cf68eca179ab9e9321300fbb94bf177451 WatchSource:0}: Error finding container 984329872cffecc2cd8667b8a40987cf68eca179ab9e9321300fbb94bf177451: Status 404 returned error can't find the container with id 984329872cffecc2cd8667b8a40987cf68eca179ab9e9321300fbb94bf177451
Jul 29 20:26:23.036249 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:23.036212    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:26:23.037580 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:23.037551    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk" event={"ID":"6fc36022-29e6-49bd-a054-ab5eb5504e60","Type":"ContainerStarted","Data":"984329872cffecc2cd8667b8a40987cf68eca179ab9e9321300fbb94bf177451"}
Jul 29 20:26:25.011069 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:25.011040    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cgmsd"
Jul 29 20:26:25.046670 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:25.046639    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk" event={"ID":"6fc36022-29e6-49bd-a054-ab5eb5504e60","Type":"ContainerStarted","Data":"b1759275c2bfc27450038f469de6789aeceb4b287c770b74225a716d290bd9af"}
Jul 29 20:26:25.063018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:25.062977    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbb7784f8-4t7wk" podStartSLOduration=1.432118701 podStartE2EDuration="4.06296339s" totalImagesPullingTime="2.630844689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:25.062246457 +0000 UTC m=+112.105938647" watchObservedRunningTime="2026-07-29 20:26:25.06296339 +0000 UTC m=+112.106655572"
Jul 29 20:26:26.218833 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:26.218803    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:26:26.219193 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:26.218851    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:26:26.219193 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:26.219108    2585 scope.go:117] "RemoveContainer" containerID="38191e14b51ebb7857f5bc9892c42fbec99302f5564edc22beeef3d694d957f5"
Jul 29 20:26:26.219266 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:26.219245    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-7lq85_openshift-console-operator(0248d8c6-87bb-4df2-8f61-f80a2c51469b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" podUID="0248d8c6-87bb-4df2-8f61-f80a2c51469b"
Jul 29 20:26:30.148251 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.148215    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:30.252419 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.252382    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:30.309187 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.309159    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"]
Jul 29 20:26:30.314456 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.314434    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"
Jul 29 20:26:30.318194 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.318175    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-b2ktb\""
Jul 29 20:26:30.318194 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.318186    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 20:26:30.324049 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.324022    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"]
Jul 29 20:26:30.330841 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.330818    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:30.398839 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.398774    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:30.428794 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.428771    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40b60c36-dfff-4809-802e-dd224bde115d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cpx9c\" (UID: \"40b60c36-dfff-4809-802e-dd224bde115d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"
Jul 29 20:26:30.529990 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.529966    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40b60c36-dfff-4809-802e-dd224bde115d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cpx9c\" (UID: \"40b60c36-dfff-4809-802e-dd224bde115d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"
Jul 29 20:26:30.532536 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.532509    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40b60c36-dfff-4809-802e-dd224bde115d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cpx9c\" (UID: \"40b60c36-dfff-4809-802e-dd224bde115d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"
Jul 29 20:26:30.625530 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.625506    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"
Jul 29 20:26:30.764991 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:30.764908    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"]
Jul 29 20:26:30.767354 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:30.767307    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b60c36_dfff_4809_802e_dd224bde115d.slice/crio-e937498e32afdbe7da58509196e0eec4e869dc89984519bca9206fafabaffa96 WatchSource:0}: Error finding container e937498e32afdbe7da58509196e0eec4e869dc89984519bca9206fafabaffa96: Status 404 returned error can't find the container with id e937498e32afdbe7da58509196e0eec4e869dc89984519bca9206fafabaffa96
Jul 29 20:26:31.066504 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:31.066476    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c" event={"ID":"40b60c36-dfff-4809-802e-dd224bde115d","Type":"ContainerStarted","Data":"e937498e32afdbe7da58509196e0eec4e869dc89984519bca9206fafabaffa96"}
Jul 29 20:26:32.241790 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:32.241762    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:32.243941 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:32.243910    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8f1b3cb-a4e4-401f-aeb2-34451e025fcf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggdgh\" (UID: \"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf\") " pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:32.249399 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:32.249378    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ggdgh"
Jul 29 20:26:32.367382 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:32.367359    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ggdgh"]
Jul 29 20:26:32.369592 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:32.369569    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f1b3cb_a4e4_401f_aeb2_34451e025fcf.slice/crio-e22d84c91118a2203976b98084a7d8a43f5515a71ae9e7db6b15d204022c7187 WatchSource:0}: Error finding container e22d84c91118a2203976b98084a7d8a43f5515a71ae9e7db6b15d204022c7187: Status 404 returned error can't find the container with id e22d84c91118a2203976b98084a7d8a43f5515a71ae9e7db6b15d204022c7187
Jul 29 20:26:32.582275 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:32.582184    2585 patch_prober.go:28] interesting pod/image-registry-6bf8fdf9cc-85tqw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 20:26:32.582275 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:32.582243    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw" podUID="2883174b-258b-4a7b-906d-237dc51e8027" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 20:26:33.075909 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:33.075820    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggdgh" event={"ID":"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf","Type":"ContainerStarted","Data":"e22d84c91118a2203976b98084a7d8a43f5515a71ae9e7db6b15d204022c7187"}
Jul 29 20:26:33.077981 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:33.077947    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c" event={"ID":"40b60c36-dfff-4809-802e-dd224bde115d","Type":"ContainerStarted","Data":"59c1be69a2df52c4519c5dfb188431a0b2436b76f1e1b8d6c58835b07043f1c1"}
Jul 29 20:26:33.078193 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:33.078175    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"
Jul 29 20:26:33.084379 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:33.084323    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c"
Jul 29 20:26:33.095009 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:33.094935    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cpx9c" podStartSLOduration=1.8452503999999998 podStartE2EDuration="3.094919654s" totalImagesPullingTime="1.249669254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:33.094778436 +0000 UTC m=+120.138470619" watchObservedRunningTime="2026-07-29 20:26:33.094919654 +0000 UTC m=+120.138611840"
Jul 29 20:26:33.392015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:33.391877    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:33.462119 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:33.462082    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:33.748325 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:33.748286    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:34.004881 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:34.004810    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:26:34.141666 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:34.141630    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:34.198160 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:34.198123    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:37.275948 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:37.275924    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:37.374035 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:37.374005    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:38.098182 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:38.098142    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggdgh" event={"ID":"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf","Type":"ContainerStarted","Data":"9ec33ee6c83200d6b3a05dcb3c0d76b7c9290d9d7df38af58960f5c9c10c848b"}
Jul 29 20:26:38.237282 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:38.237249    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:39.519348 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:39.519296    2585 scope.go:117] "RemoveContainer" containerID="38191e14b51ebb7857f5bc9892c42fbec99302f5564edc22beeef3d694d957f5"
Jul 29 20:26:39.519772 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:39.519548    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-7lq85_openshift-console-operator(0248d8c6-87bb-4df2-8f61-f80a2c51469b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" podUID="0248d8c6-87bb-4df2-8f61-f80a2c51469b"
Jul 29 20:26:40.518171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.518131    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:40.753115 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.753029    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:40.814809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.814778    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"]
Jul 29 20:26:40.819341 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.819307    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:40.823153 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.823129    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-779lt\""
Jul 29 20:26:40.823458 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.823363    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 29 20:26:40.825930 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.824579    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 29 20:26:40.825930 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.824910    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 20:26:40.825930 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.824919    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 20:26:40.825930 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.825159    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 20:26:40.825930 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.825438    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c2zzj"]
Jul 29 20:26:40.825930 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.825661    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 29 20:26:40.828935 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.828913    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.833424 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.833389    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 20:26:40.833626 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.833597    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 20:26:40.834113 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.833922    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t564f\""
Jul 29 20:26:40.834113 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.833939    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 20:26:40.843399 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.842623    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"]
Jul 29 20:26:40.899953 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.899931    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:40.909952 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.909926    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bdfb728-4abe-469f-9c5b-9966b0f8f429-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:40.910075 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.909965    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f555377d-dc85-4e77-8da4-a2a4162dc45e-sys\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.910075 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910059    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-textfile\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.910187 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910099    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:40.910187 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910131    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:40.910187 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910158    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbh7b\" (UniqueName: \"kubernetes.io/projected/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-api-access-nbh7b\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:40.910352 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910188    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.910352 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910211    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:40.910352 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910237    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-tls\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.910352 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910296    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f555377d-dc85-4e77-8da4-a2a4162dc45e-metrics-client-ca\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.910555 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910384    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqm7c\" (UniqueName: \"kubernetes.io/projected/f555377d-dc85-4e77-8da4-a2a4162dc45e-kube-api-access-fqm7c\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.910555 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910456    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.910555 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910497    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3bdfb728-4abe-469f-9c5b-9966b0f8f429-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:40.910555 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910546    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f555377d-dc85-4e77-8da4-a2a4162dc45e-root\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.910745 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.910638    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-wtmp\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:40.975894 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:40.975866    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:41.011924 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.011901    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.012039 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.011936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbh7b\" (UniqueName: \"kubernetes.io/projected/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-api-access-nbh7b\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.012039 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.011964    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012039 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.011987    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.012039 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012011    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-tls\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012054    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f555377d-dc85-4e77-8da4-a2a4162dc45e-metrics-client-ca\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012087    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqm7c\" (UniqueName: \"kubernetes.io/projected/f555377d-dc85-4e77-8da4-a2a4162dc45e-kube-api-access-fqm7c\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012115    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3bdfb728-4abe-469f-9c5b-9966b0f8f429-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.012240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012166    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f555377d-dc85-4e77-8da4-a2a4162dc45e-root\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012199    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-wtmp\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bdfb728-4abe-469f-9c5b-9966b0f8f429-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.012604 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012258    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f555377d-dc85-4e77-8da4-a2a4162dc45e-sys\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012604 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012297    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-textfile\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012604 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012325    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.012753 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012626    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f555377d-dc85-4e77-8da4-a2a4162dc45e-sys\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.012753 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.012670    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-wtmp\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.013051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.013031    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.013221 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.013195    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.013273 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.013243    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bdfb728-4abe-469f-9c5b-9966b0f8f429-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.013323 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.013278    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-textfile\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.013323 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.013303    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f555377d-dc85-4e77-8da4-a2a4162dc45e-root\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.013542 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.013525    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3bdfb728-4abe-469f-9c5b-9966b0f8f429-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.013663 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:41.013641    2585 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 29 20:26:41.013729 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:41.013709    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-tls podName:3bdfb728-4abe-469f-9c5b-9966b0f8f429 nodeName:}" failed. No retries permitted until 2026-07-29 20:26:41.513692136 +0000 UTC m=+128.557384303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-vzwdj" (UID: "3bdfb728-4abe-469f-9c5b-9966b0f8f429") : secret "kube-state-metrics-tls" not found
Jul 29 20:26:41.014362 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.014320    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f555377d-dc85-4e77-8da4-a2a4162dc45e-metrics-client-ca\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.016143 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.016050    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.016143 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.016108    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.016466 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.016445    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f555377d-dc85-4e77-8da4-a2a4162dc45e-node-exporter-tls\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.023176 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.023107    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqm7c\" (UniqueName: \"kubernetes.io/projected/f555377d-dc85-4e77-8da4-a2a4162dc45e-kube-api-access-fqm7c\") pod \"node-exporter-c2zzj\" (UID: \"f555377d-dc85-4e77-8da4-a2a4162dc45e\") " pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.023828 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.023792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbh7b\" (UniqueName: \"kubernetes.io/projected/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-api-access-nbh7b\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.116665 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.116634    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggdgh" event={"ID":"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf","Type":"ContainerStarted","Data":"b2f28f77ec0e414974543254a8178c688b179e8a0049b066b69d805578f56b42"}
Jul 29 20:26:41.117673 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.117646    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:41.146270 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.146251    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c2zzj"
Jul 29 20:26:41.159895 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:41.159871    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf555377d_dc85_4e77_8da4_a2a4162dc45e.slice/crio-36192c9061073b722f28daf8a6dff09161e69f888db7c1d3346e31059a63c45a WatchSource:0}: Error finding container 36192c9061073b722f28daf8a6dff09161e69f888db7c1d3346e31059a63c45a: Status 404 returned error can't find the container with id 36192c9061073b722f28daf8a6dff09161e69f888db7c1d3346e31059a63c45a
Jul 29 20:26:41.221353 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.221301    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:41.385420 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.385349    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:41.517069 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.517037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.519736 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.519691    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:41.520083 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.520061    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bdfb728-4abe-469f-9c5b-9966b0f8f429-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vzwdj\" (UID: \"3bdfb728-4abe-469f-9c5b-9966b0f8f429\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.724235 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.724209    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:41.736521 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.736301    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"
Jul 29 20:26:41.814302 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.814268    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 20:26:41.820195 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.820143    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.821772 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.821733    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:41.826645 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.826391    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 20:26:41.828346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.827748    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 20:26:41.828346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.827905    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 20:26:41.828346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.827936    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 20:26:41.828346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.828047    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 20:26:41.828346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.828205    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-spqhl\""
Jul 29 20:26:41.828346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.828257    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 20:26:41.828703 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.828209    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 20:26:41.828703 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.828453    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 20:26:41.828703 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.828619    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 20:26:41.849741 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.849717    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 20:26:41.903104 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.903029    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj"]
Jul 29 20:26:41.905972 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:41.905944    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdfb728_4abe_469f_9c5b_9966b0f8f429.slice/crio-fd6f66fc3940b342fe8e396dfa078cb2ae25d2eaf6c3eed981c445c4f110faf6 WatchSource:0}: Error finding container fd6f66fc3940b342fe8e396dfa078cb2ae25d2eaf6c3eed981c445c4f110faf6: Status 404 returned error can't find the container with id fd6f66fc3940b342fe8e396dfa078cb2ae25d2eaf6c3eed981c445c4f110faf6
Jul 29 20:26:41.907972 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.907942    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:41.920586 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.920564    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.920690 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.920600    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.920690 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.920629    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-web-config\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.920797 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.920716    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.920797 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.920770    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.920947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.920926    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-volume\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.921004 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.920960    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.921058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.921007    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.921213 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.921092    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4p6\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-kube-api-access-4f4p6\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.921312 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.921229    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.921312 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.921259    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.921312 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.921283    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:41.921452 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:41.921312    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-out\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022077 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.021997    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-volume\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022077 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022044    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022282 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022195    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022282 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022246    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4p6\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-kube-api-access-4f4p6\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022282 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022274    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022470 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022470 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022349    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022470 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022405    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-out\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022470 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022459    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022668 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022496    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022668 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022542    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-web-config\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022668 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022668 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.022934 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.022913    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.024459 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.023167    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.024459 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:26:42.023993    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-trusted-ca-bundle podName:30a086dd-07a4-4ebb-bce8-cbfb0aff030f nodeName:}" failed. No retries permitted until 2026-07-29 20:26:42.523970398 +0000 UTC m=+129.567662563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f") : configmap references non-existent config key: ca-bundle.crt
Jul 29 20:26:42.025526 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.025483    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.026205 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.026089    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.026874 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.026835    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-out\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.027434 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.027381    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.027434 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.027408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-volume\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.027775 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.027755    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.028282 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.028261    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-web-config\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.028870 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.028848    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.029765 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.029721    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.034482 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.034458    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4p6\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-kube-api-access-4f4p6\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.123715 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.123669    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2zzj" event={"ID":"f555377d-dc85-4e77-8da4-a2a4162dc45e","Type":"ContainerStarted","Data":"36192c9061073b722f28daf8a6dff09161e69f888db7c1d3346e31059a63c45a"}
Jul 29 20:26:42.125449 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.125380    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj" event={"ID":"3bdfb728-4abe-469f-9c5b-9966b0f8f429","Type":"ContainerStarted","Data":"fd6f66fc3940b342fe8e396dfa078cb2ae25d2eaf6c3eed981c445c4f110faf6"}
Jul 29 20:26:42.229304 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.229273    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:42.330620 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.330549    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:42.527371 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.527319    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.528726 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.528678    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.734512 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.734474    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:26:42.899905 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:42.899873    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 20:26:43.131606 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.131527    2585 generic.go:356] "Generic (PLEG): container finished" podID="f555377d-dc85-4e77-8da4-a2a4162dc45e" containerID="c6a0cd8c3dcb583b89762bfe503cc9be0533e45c940966fc5072166d2c56f4e7" exitCode=0
Jul 29 20:26:43.131795 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.131603    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2zzj" event={"ID":"f555377d-dc85-4e77-8da4-a2a4162dc45e","Type":"ContainerDied","Data":"c6a0cd8c3dcb583b89762bfe503cc9be0533e45c940966fc5072166d2c56f4e7"}
Jul 29 20:26:43.141541 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:43.141521    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a086dd_07a4_4ebb_bce8_cbfb0aff030f.slice/crio-b0ea1bec0bbf514ec670690f5ca6e6f60cfd2057191cdb7bc78f4965339e22a5 WatchSource:0}: Error finding container b0ea1bec0bbf514ec670690f5ca6e6f60cfd2057191cdb7bc78f4965339e22a5: Status 404 returned error can't find the container with id b0ea1bec0bbf514ec670690f5ca6e6f60cfd2057191cdb7bc78f4965339e22a5
Jul 29 20:26:43.231707 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.231665    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:26:43.234486 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.234453    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94bf5594-165a-4efb-ba8c-a628ea34eabf-metrics-certs\") pod \"network-metrics-daemon-7sqwj\" (UID: \"94bf5594-165a-4efb-ba8c-a628ea34eabf\") " pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:26:43.341339 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.341296    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7tqs\""
Jul 29 20:26:43.348696 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.348672    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqwj"
Jul 29 20:26:43.478003 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.477970    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"]
Jul 29 20:26:43.580406 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.580367    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:43.673378 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.673316    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:43.774830 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.774745    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:43.825443 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.825416    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7cf4864fc6-z597l"]
Jul 29 20:26:43.838458 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.838434    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:43.841650 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.841625    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 29 20:26:43.841786 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.841693    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 29 20:26:43.842258 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.842234    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rtkgx\""
Jul 29 20:26:43.842489 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.842471    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6i7d11970t4vg\""
Jul 29 20:26:43.842697 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.842679    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 29 20:26:43.842999 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.842978    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 29 20:26:43.846421 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.843179    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 29 20:26:43.852310 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.852274    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7cf4864fc6-z597l"]
Jul 29 20:26:43.938149 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.938123    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:43.938553 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.938186    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:43.938553 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.938221    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:43.938553 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.938265    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-metrics-client-ca\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:43.938553 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.938373    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-tls\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:43.938553 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.938429    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926sw\" (UniqueName: \"kubernetes.io/projected/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-kube-api-access-926sw\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:43.938553 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.938509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:43.938847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.938586    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-grpc-tls\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:43.988939 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:43.988907    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:44.039465 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.039399    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-grpc-tls\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.039574 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.039546    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.039639 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.039589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.039639 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.039632    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.039741 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.039659    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-metrics-client-ca\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.039741 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.039696    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-tls\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.039847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.039739    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926sw\" (UniqueName: \"kubernetes.io/projected/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-kube-api-access-926sw\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.039847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.039782    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.040785 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.040732    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-metrics-client-ca\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.042718 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.042692    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-grpc-tls\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.043037 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.043014    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.043245 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.043224    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-tls\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.043546 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.043528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.043894 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.043857    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.044978 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.044953    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.048871 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.048848    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-926sw\" (UniqueName: \"kubernetes.io/projected/a30a62ac-86d3-42e6-8cdf-a932c07ebdc0-kube-api-access-926sw\") pod \"thanos-querier-7cf4864fc6-z597l\" (UID: \"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0\") " pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.089714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.089677    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:44.136897 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.136866    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerStarted","Data":"b0ea1bec0bbf514ec670690f5ca6e6f60cfd2057191cdb7bc78f4965339e22a5"}
Jul 29 20:26:44.160323 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.160278    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:44.189876 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:44.189838    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:45.040532 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.040500    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7cf4864fc6-z597l"]
Jul 29 20:26:45.060493 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.060424    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7sqwj"]
Jul 29 20:26:45.068243 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:45.067770    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bf5594_165a_4efb_ba8c_a628ea34eabf.slice/crio-9c80201b3666296ce050c5a21c9e9619d5e9c33ccf3f24789d6e59fd575bb2ab WatchSource:0}: Error finding container 9c80201b3666296ce050c5a21c9e9619d5e9c33ccf3f24789d6e59fd575bb2ab: Status 404 returned error can't find the container with id 9c80201b3666296ce050c5a21c9e9619d5e9c33ccf3f24789d6e59fd575bb2ab
Jul 29 20:26:45.145826 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.145487    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggdgh" event={"ID":"b8f1b3cb-a4e4-401f-aeb2-34451e025fcf","Type":"ContainerStarted","Data":"d092fc54785aa2ff91e21cd88b0bc7fc53ffd147c01db7811a2529e4e0d0f41d"}
Jul 29 20:26:45.150569 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.150511    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2zzj" event={"ID":"f555377d-dc85-4e77-8da4-a2a4162dc45e","Type":"ContainerStarted","Data":"1ddaff2e4977773983f37e1b1c8b2e8d4350832b13f38ecedbe98bbc40c407b2"}
Jul 29 20:26:45.150649 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.150577    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2zzj" event={"ID":"f555377d-dc85-4e77-8da4-a2a4162dc45e","Type":"ContainerStarted","Data":"cfa8a238e9193443be62878a9924fa92b6d7880d0706c83ceaff240a578dcfb8"}
Jul 29 20:26:45.153670 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.153584    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l" event={"ID":"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0","Type":"ContainerStarted","Data":"cee0323e0c1d871a26987c88531587e30463d81bf17394bade134a34653e23e1"}
Jul 29 20:26:45.157143 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.157079    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj" event={"ID":"3bdfb728-4abe-469f-9c5b-9966b0f8f429","Type":"ContainerStarted","Data":"d82a082ae3dd3dc5a35046fdfd8a7d364f5965b49c58d40b7353f91db2b816f6"}
Jul 29 20:26:45.157453 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.157264    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj" event={"ID":"3bdfb728-4abe-469f-9c5b-9966b0f8f429","Type":"ContainerStarted","Data":"d0d7ca462ed6e8a05c4326aada8794e9540c0521f37d75250aa7c7d5e5b3e543"}
Jul 29 20:26:45.157453 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.157303    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj" event={"ID":"3bdfb728-4abe-469f-9c5b-9966b0f8f429","Type":"ContainerStarted","Data":"6f52adffd861fb756083c86a26b179a08d1bd05779ffb51069a7d91fb989ca57"}
Jul 29 20:26:45.159426 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.159373    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7sqwj" event={"ID":"94bf5594-165a-4efb-ba8c-a628ea34eabf","Type":"ContainerStarted","Data":"9c80201b3666296ce050c5a21c9e9619d5e9c33ccf3f24789d6e59fd575bb2ab"}
Jul 29 20:26:45.192204 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.192151    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vzwdj" podStartSLOduration=2.214569241 podStartE2EDuration="5.192136005s" totalImagesPullingTime="2.977566764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:45.191171763 +0000 UTC m=+132.234863947" watchObservedRunningTime="2026-07-29 20:26:45.192136005 +0000 UTC m=+132.235828188"
Jul 29 20:26:45.192620 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.192592    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ggdgh" podStartSLOduration=32.792672523 podStartE2EDuration="45.19258366s" totalImagesPullingTime="12.399911137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:26:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:45.166213619 +0000 UTC m=+132.209905803" watchObservedRunningTime="2026-07-29 20:26:45.19258366 +0000 UTC m=+132.236275844"
Jul 29 20:26:45.220547 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.220507    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c2zzj" podStartSLOduration=4.085061405 podStartE2EDuration="5.220493214s" totalImagesPullingTime="1.05940107s" totalInitContainerRuntime="76.030739ms" isStatefulPod=true podCreationTimestamp="2026-07-29 20:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:45.219196507 +0000 UTC m=+132.262888691" watchObservedRunningTime="2026-07-29 20:26:45.220493214 +0000 UTC m=+132.264185414"
Jul 29 20:26:45.568958 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.568915    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:45.663899 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:45.663851    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:46.166173 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:46.166133    2585 generic.go:356] "Generic (PLEG): container finished" podID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerID="d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da" exitCode=0
Jul 29 20:26:46.166642 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:46.166424    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerDied","Data":"d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da"}
Jul 29 20:26:46.354765 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:46.354683    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:46.421383 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:46.421295    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:47.046763 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.046728    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 20:26:47.047919 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.047889    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:47.051282 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.051265    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.055274 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.054299    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 20:26:47.055274 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.054528    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 20:26:47.055274 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.054727    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-prk6z\""
Jul 29 20:26:47.055274 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.054935    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 20:26:47.055274 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.055029    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 20:26:47.055274 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.055125    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 20:26:47.055274 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.055258    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 20:26:47.056361 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.056206    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 20:26:47.056361 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.056232    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 20:26:47.057640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.056497    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 20:26:47.057640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.056565    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 20:26:47.057640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.056690    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d7ts5iqpsdi2t\""
Jul 29 20:26:47.057640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.056692    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 20:26:47.057640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.056810    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 20:26:47.057640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.056998    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 20:26:47.057640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.057373    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 20:26:47.058027 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.057961    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 20:26:47.074015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.073989    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 20:26:47.168947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.168916    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-config-out\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.168957    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.168994    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-config\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169063    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169127    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169161    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169191    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169220    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169248    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-web-config\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169300    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169342    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169366    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.169411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169416    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.170063 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169442    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.170063 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169465    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.170063 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169495    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.170063 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169528    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.170063 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169554    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jbq\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-kube-api-access-78jbq\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.170063 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.169578    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.173590 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.173558    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7sqwj" event={"ID":"94bf5594-165a-4efb-ba8c-a628ea34eabf","Type":"ContainerStarted","Data":"3479f800ca79ec8770d6965cedeb45128a8b80f9abf98fc429f68a5395ca8ee1"}
Jul 29 20:26:47.173706 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.173611    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7sqwj" event={"ID":"94bf5594-165a-4efb-ba8c-a628ea34eabf","Type":"ContainerStarted","Data":"2b2f1a3c950f1da5f36bf48bad7625cf4ad5fd358f1f76c1f7a8233ba6d3579f"}
Jul 29 20:26:47.192924 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.192867    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7sqwj" podStartSLOduration=131.941745395 podStartE2EDuration="2m13.192848226s" totalImagesPullingTime="1.251102831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:47.191188448 +0000 UTC m=+134.234880633" watchObservedRunningTime="2026-07-29 20:26:47.192848226 +0000 UTC m=+134.236540410"
Jul 29 20:26:47.270240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.270184    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-config-out\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.270240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.270222    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.270466 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.270255    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-config\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.270466 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.270310    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.270893 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.270868    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.270984 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.270922    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.270984 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.270952    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.270984 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.270976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271034    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-web-config\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271061    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271090    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271112    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271381 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271381 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271381 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271197    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271381 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271234    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271381 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271274    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271381 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271319    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jbq\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-kube-api-access-78jbq\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271381 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271363    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.271989 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.271961    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.274103 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.273041    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.274103 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.273567    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.274103 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.273966    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.274695 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.274671    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.274940 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.274916    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.275017 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.274988    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-config\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.275159 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.275131    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-config-out\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.276573 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.275581    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.276573 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.275802    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.276573 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.275900    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-web-config\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.276573 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.276143    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.276573 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.276448    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.276573 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.276535    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.276971 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.276598    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.276971 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.276606    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.277506 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.277488    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.277885 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.277866    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.278655 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.278636    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.288099 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.288080    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jbq\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-kube-api-access-78jbq\") pod \"prometheus-k8s-0\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.336382 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.336276    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:47.363604 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.363578    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:47.470564 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:47.470528    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:48.134416 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:48.128871    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 20:26:48.137504 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:26:48.137389    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66618f3e_62a8_484b_9da3_0f0e404e3667.slice/crio-12569173e0b52bf05a0104d558344885a7dc63b5796c18ac14956cfd8ce65047 WatchSource:0}: Error finding container 12569173e0b52bf05a0104d558344885a7dc63b5796c18ac14956cfd8ce65047: Status 404 returned error can't find the container with id 12569173e0b52bf05a0104d558344885a7dc63b5796c18ac14956cfd8ce65047
Jul 29 20:26:48.180499 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:48.180237    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerStarted","Data":"6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072"}
Jul 29 20:26:48.180499 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:48.180324    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerStarted","Data":"239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c"}
Jul 29 20:26:48.183102 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:48.182962    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l" event={"ID":"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0","Type":"ContainerStarted","Data":"01ab6af2c440dd672eb0bd58e684c27b6d4af8e5138963fe901dc9ca9f5c56ad"}
Jul 29 20:26:48.183102 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:48.183015    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l" event={"ID":"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0","Type":"ContainerStarted","Data":"20c2ae65dae9da9c372c3a4562bddf73bd0215b8e9e1ea16448b502c1a9c832d"}
Jul 29 20:26:48.184861 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:48.184840    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerStarted","Data":"12569173e0b52bf05a0104d558344885a7dc63b5796c18ac14956cfd8ce65047"}
Jul 29 20:26:49.192203 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:49.192179    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerStarted","Data":"0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c"}
Jul 29 20:26:49.192578 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:49.192256    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerStarted","Data":"d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814"}
Jul 29 20:26:49.192578 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:49.192297    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerStarted","Data":"e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb"}
Jul 29 20:26:49.194929 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:49.194900    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l" event={"ID":"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0","Type":"ContainerStarted","Data":"fd4d0b780c436b1e55de06b675d954824b819654d1c15e8f4914df01a1f4720c"}
Jul 29 20:26:49.197006 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:49.196981    2585 generic.go:356] "Generic (PLEG): container finished" podID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerID="47274529b74b8cf169e31ee2b0d8d4964cf0e843e0337bee2451d2e260c89de5" exitCode=0
Jul 29 20:26:49.197086 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:49.197009    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerDied","Data":"47274529b74b8cf169e31ee2b0d8d4964cf0e843e0337bee2451d2e260c89de5"}
Jul 29 20:26:50.208645 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:50.208609    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerStarted","Data":"b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465"}
Jul 29 20:26:50.211389 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:50.211364    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l" event={"ID":"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0","Type":"ContainerStarted","Data":"eb09ac838bd5da2780b1c08711fde23eb5e7ff8463cfa1a8361d69de9bcb53de"}
Jul 29 20:26:50.211495 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:50.211421    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l" event={"ID":"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0","Type":"ContainerStarted","Data":"9bd9e66ddfc9647b93ed3ae2bd9d8135f94c74805bada094435aac2534788790"}
Jul 29 20:26:50.211495 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:50.211451    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:50.211495 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:50.211475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l" event={"ID":"a30a62ac-86d3-42e6-8cdf-a932c07ebdc0","Type":"ContainerStarted","Data":"21ab5465f1ab00cac71f507cc2a3733e5cf00a13bf2b3c2b703890dd9c2e00a7"}
Jul 29 20:26:50.239456 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:50.239414    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.118942931 podStartE2EDuration="9.239400803s" totalImagesPullingTime="5.101280145s" totalInitContainerRuntime="19.177727ms" isStatefulPod=false podCreationTimestamp="2026-07-29 20:26:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:50.238549864 +0000 UTC m=+137.282242058" watchObservedRunningTime="2026-07-29 20:26:50.239400803 +0000 UTC m=+137.283093035"
Jul 29 20:26:50.262782 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:50.262734    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l" podStartSLOduration=3.415098054 podStartE2EDuration="7.26271466s" totalImagesPullingTime="3.847616606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:26:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:50.262399849 +0000 UTC m=+137.306092034" watchObservedRunningTime="2026-07-29 20:26:50.26271466 +0000 UTC m=+137.306406844"
Jul 29 20:26:52.200986 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:52.200936    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:52.306285 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:52.306243    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:53.229308 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:53.229273    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerStarted","Data":"b971a0c92d4b2272419b2031a1fe761109058dfa18e957bbb906d4fbe3ebd87a"}
Jul 29 20:26:53.229698 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:53.229362    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerStarted","Data":"8748ab872a7d9b59d463b616fdddea6241d268beacb72df7265cb9fb68fdc7c8"}
Jul 29 20:26:53.229698 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:53.229401    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerStarted","Data":"b8b3f16041603fc58e37488e6857e2f6a8efc7535ee9f6cc51773f62a3a0e521"}
Jul 29 20:26:53.229698 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:53.229429    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerStarted","Data":"d83cd1e28b751960ee9e5cd0b5acffe37205ea34de4c774b7aaa4d12400c2105"}
Jul 29 20:26:53.229698 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:53.229452    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerStarted","Data":"75223a741b8b3f8799de6ef4fc184a51ef2c175574d962da94920693f61bbac5"}
Jul 29 20:26:53.229698 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:53.229486    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerStarted","Data":"a4871d493ae0c26bd8f3859ec8d5b7a523a1d376868a05cef9991118254ab7bc"}
Jul 29 20:26:53.265809 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:53.265756    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.006643469 podStartE2EDuration="6.265741209s" totalImagesPullingTime="3.2388879s" totalInitContainerRuntime="20.20984ms" isStatefulPod=false podCreationTimestamp="2026-07-29 20:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:53.264772126 +0000 UTC m=+140.308464323" watchObservedRunningTime="2026-07-29 20:26:53.265741209 +0000 UTC m=+140.309433391"
Jul 29 20:26:54.518853 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:54.518823    2585 scope.go:117] "RemoveContainer" containerID="38191e14b51ebb7857f5bc9892c42fbec99302f5564edc22beeef3d694d957f5"
Jul 29 20:26:55.238596 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.238570    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:26:55.238777 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.238621    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" event={"ID":"0248d8c6-87bb-4df2-8f61-f80a2c51469b","Type":"ContainerStarted","Data":"f167180cb97c565110077332c9f97eb84e95d2461bba2631cebe2cdfe29cf27e"}
Jul 29 20:26:55.238999 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.238975    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:26:55.243646 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.243620    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85"
Jul 29 20:26:55.264794 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.264749    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-7lq85" podStartSLOduration=57.435725245 podStartE2EDuration="1m0.264736435s" totalImagesPullingTime="2.82901119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:26:55.263571414 +0000 UTC m=+142.307263596" watchObservedRunningTime="2026-07-29 20:26:55.264736435 +0000 UTC m=+142.308428657"
Jul 29 20:26:55.272172 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.272146    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:55.373573 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.373546    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:55.526125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.526047    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:55.707473 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.707442    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:55.774495 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:55.774464    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:56.104643 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:56.104607    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:56.168641 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:56.168599    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:56.223800 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:56.223775    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7cf4864fc6-z597l"
Jul 29 20:26:56.323545 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:56.323511    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:56.539022 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:56.538987    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:56.724997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:56.724949    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:56.822396 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:56.822302    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:56.921130 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:56.921097    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:57.363947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:57.363918    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:26:58.760800 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:58.760762    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:58.827290 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:58.827257    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:59.114385 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:59.114290    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:26:59.223575 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:26:59.223535    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:01.296745 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:01.296710    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:01.358704 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:01.358671    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:06.950120 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:06.950072    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:07.050940 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:07.050908    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:07.151827 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:07.151799    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:08.503941 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.503881    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw" podUID="2883174b-258b-4a7b-906d-237dc51e8027" containerName="registry" containerID="cri-o://47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266" gracePeriod=30
Jul 29 20:27:08.779592 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.779525    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:27:08.867879 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.867851    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44fnc\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-kube-api-access-44fnc\") pod \"2883174b-258b-4a7b-906d-237dc51e8027\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") "
Jul 29 20:27:08.868058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.867902    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-registry-certificates\") pod \"2883174b-258b-4a7b-906d-237dc51e8027\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") "
Jul 29 20:27:08.868058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.867932    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-image-registry-private-configuration\") pod \"2883174b-258b-4a7b-906d-237dc51e8027\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") "
Jul 29 20:27:08.868058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.868001    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") pod \"2883174b-258b-4a7b-906d-237dc51e8027\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") "
Jul 29 20:27:08.868058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.868018    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-trusted-ca\") pod \"2883174b-258b-4a7b-906d-237dc51e8027\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") "
Jul 29 20:27:08.868058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.868038    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2883174b-258b-4a7b-906d-237dc51e8027-ca-trust-extracted\") pod \"2883174b-258b-4a7b-906d-237dc51e8027\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") "
Jul 29 20:27:08.868058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.868057    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-bound-sa-token\") pod \"2883174b-258b-4a7b-906d-237dc51e8027\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") "
Jul 29 20:27:08.868767 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.868083    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-installation-pull-secrets\") pod \"2883174b-258b-4a7b-906d-237dc51e8027\" (UID: \"2883174b-258b-4a7b-906d-237dc51e8027\") "
Jul 29 20:27:08.868767 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.868486    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2883174b-258b-4a7b-906d-237dc51e8027" (UID: "2883174b-258b-4a7b-906d-237dc51e8027"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:27:08.868767 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.868731    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2883174b-258b-4a7b-906d-237dc51e8027" (UID: "2883174b-258b-4a7b-906d-237dc51e8027"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:27:08.870924 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.870866    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-kube-api-access-44fnc" (OuterVolumeSpecName: "kube-api-access-44fnc") pod "2883174b-258b-4a7b-906d-237dc51e8027" (UID: "2883174b-258b-4a7b-906d-237dc51e8027"). InnerVolumeSpecName "kube-api-access-44fnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:27:08.870924 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.870887    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2883174b-258b-4a7b-906d-237dc51e8027" (UID: "2883174b-258b-4a7b-906d-237dc51e8027"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:27:08.871103 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.870934    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2883174b-258b-4a7b-906d-237dc51e8027" (UID: "2883174b-258b-4a7b-906d-237dc51e8027"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:27:08.871103 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.870967    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2883174b-258b-4a7b-906d-237dc51e8027" (UID: "2883174b-258b-4a7b-906d-237dc51e8027"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:27:08.871103 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.871018    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2883174b-258b-4a7b-906d-237dc51e8027" (UID: "2883174b-258b-4a7b-906d-237dc51e8027"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:27:08.877708 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.877682    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2883174b-258b-4a7b-906d-237dc51e8027-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2883174b-258b-4a7b-906d-237dc51e8027" (UID: "2883174b-258b-4a7b-906d-237dc51e8027"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:27:08.969074 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.969050    2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-registry-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:27:08.969074 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.969071    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-trusted-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:27:08.969182 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.969080    2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2883174b-258b-4a7b-906d-237dc51e8027-ca-trust-extracted\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:27:08.969182 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.969089    2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-bound-sa-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:27:08.969182 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.969097    2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-installation-pull-secrets\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:27:08.969182 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.969105    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44fnc\" (UniqueName: \"kubernetes.io/projected/2883174b-258b-4a7b-906d-237dc51e8027-kube-api-access-44fnc\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:27:08.969182 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.969114    2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2883174b-258b-4a7b-906d-237dc51e8027-registry-certificates\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:27:08.969182 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:08.969123    2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2883174b-258b-4a7b-906d-237dc51e8027-image-registry-private-configuration\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:27:09.298256 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.298227    2585 generic.go:356] "Generic (PLEG): container finished" podID="2883174b-258b-4a7b-906d-237dc51e8027" containerID="47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266" exitCode=0
Jul 29 20:27:09.298453 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.298270    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw" event={"ID":"2883174b-258b-4a7b-906d-237dc51e8027","Type":"ContainerDied","Data":"47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266"}
Jul 29 20:27:09.298453 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.298284    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"
Jul 29 20:27:09.298453 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.298355    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bf8fdf9cc-85tqw" event={"ID":"2883174b-258b-4a7b-906d-237dc51e8027","Type":"ContainerDied","Data":"8a9429f1ade29edfbba21097d4f5ae9e23bf44f4017273dca5329deaa1d9f4ec"}
Jul 29 20:27:09.298453 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.298380    2585 scope.go:117] "RemoveContainer" containerID="47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266"
Jul 29 20:27:09.307192 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.307171    2585 scope.go:117] "RemoveContainer" containerID="47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266"
Jul 29 20:27:09.307489 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:27:09.307470    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266\": container with ID starting with 47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266 not found: ID does not exist" containerID="47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266"
Jul 29 20:27:09.307547 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.307495    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266"} err="failed to get container status \"47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266\": rpc error: code = NotFound desc = could not find container \"47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266\": container with ID starting with 47fae7c1dc19af03abe165ac06344f51c14dbe5a064f901f370f43696387d266 not found: ID does not exist"
Jul 29 20:27:09.321225 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.321202    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"]
Jul 29 20:27:09.323589 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.323569    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bf8fdf9cc-85tqw"]
Jul 29 20:27:09.523092 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:09.523063    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2883174b-258b-4a7b-906d-237dc51e8027" path="/var/lib/kubelet/pods/2883174b-258b-4a7b-906d-237dc51e8027/volumes"
Jul 29 20:27:11.134580 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:11.134542    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:11.200443 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:11.200413    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:14.080970 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:14.080935    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:14.157200 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:14.157168    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:14.748596 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:14.748565    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:14.823873 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:14.823840    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:15.325045 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:15.324968    2585 generic.go:356] "Generic (PLEG): container finished" podID="e7bc6de2-919c-4f33-98ea-1a838956bd8f" containerID="4dc33391f3497c18523abd8d9cc218a652f47ac528d35200bc38fa44ff7f1f6a" exitCode=0
Jul 29 20:27:15.325434 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:15.325046    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl" event={"ID":"e7bc6de2-919c-4f33-98ea-1a838956bd8f","Type":"ContainerDied","Data":"4dc33391f3497c18523abd8d9cc218a652f47ac528d35200bc38fa44ff7f1f6a"}
Jul 29 20:27:15.325434 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:15.325391    2585 scope.go:117] "RemoveContainer" containerID="4dc33391f3497c18523abd8d9cc218a652f47ac528d35200bc38fa44ff7f1f6a"
Jul 29 20:27:16.331255 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:16.331218    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl" event={"ID":"e7bc6de2-919c-4f33-98ea-1a838956bd8f","Type":"ContainerStarted","Data":"5c83ea60ff0a7aad6662adb4419053499976da55d8ef34d3904f1f52ff18be70"}
Jul 29 20:27:18.825222 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:18.825177    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:18.925645 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:18.925610    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:20.942111 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:20.942071    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:21.005405 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:21.005376    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:21.541934 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:21.541896    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:21.643281 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:21.643248    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:23.213762 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:23.213723    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:23.277960 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:23.277928    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:25.440363 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:25.440314    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:25.490976 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:25.490948    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:25.577756 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:25.577732    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:26.463957 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:26.463879    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:26.528904 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:26.528876    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:29.086466 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:29.086419    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:29.186707 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:29.186674    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:33.180523 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:33.180485    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:33.281386 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:33.281355    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:42.767695 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:42.767656    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:42.823694 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:42.823668    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:44.144429 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:44.144397    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:44.272278 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:44.272247    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:47.364813 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:47.364775    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:27:47.383696 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:47.383672    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:27:47.468766 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:47.468740    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:27:56.884603 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:56.884564    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:56.948899 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:56.948865    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:57.339140 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:57.339097    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:57.393768 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:57.393730    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:57.486041 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:57.486003    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:57.587925 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:57.587889    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:58.113544 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:58.113508    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:27:58.181364 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:27:58.181320    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:28:01.107316 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.107279    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 20:28:01.107680 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.107580    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="alertmanager" containerID="cri-o://239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c" gracePeriod=120
Jul 29 20:28:01.107680 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.107640    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy-web" containerID="cri-o://e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb" gracePeriod=120
Jul 29 20:28:01.107680 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.107666    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="prom-label-proxy" containerID="cri-o://b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465" gracePeriod=120
Jul 29 20:28:01.107847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.107684    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="config-reloader" containerID="cri-o://6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072" gracePeriod=120
Jul 29 20:28:01.107847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.107741    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy" containerID="cri-o://d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814" gracePeriod=120
Jul 29 20:28:01.107847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.107641    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy-metric" containerID="cri-o://0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c" gracePeriod=120
Jul 29 20:28:01.510215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.510181    2585 generic.go:356] "Generic (PLEG): container finished" podID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerID="b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465" exitCode=0
Jul 29 20:28:01.510215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.510211    2585 generic.go:356] "Generic (PLEG): container finished" podID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerID="d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814" exitCode=0
Jul 29 20:28:01.510215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.510219    2585 generic.go:356] "Generic (PLEG): container finished" podID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerID="6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072" exitCode=0
Jul 29 20:28:01.510471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.510227    2585 generic.go:356] "Generic (PLEG): container finished" podID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerID="239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c" exitCode=0
Jul 29 20:28:01.510471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.510245    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerDied","Data":"b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465"}
Jul 29 20:28:01.510471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.510303    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerDied","Data":"d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814"}
Jul 29 20:28:01.510471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.510347    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerDied","Data":"6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072"}
Jul 29 20:28:01.510471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:01.510374    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerDied","Data":"239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c"}
Jul 29 20:28:02.345775 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.345747    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.470262 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470237    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-main-db\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.470433 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470269    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-web\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.470433 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470290    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.470433 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470315    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f4p6\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-kube-api-access-4f4p6\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.470433 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470349    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.470433 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470372    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-tls-assets\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.470433 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470395    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-cluster-tls-config\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.470752 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470611    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:28:02.470752 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470693    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-main-tls\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.470961 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470938    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-trusted-ca-bundle\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.471046 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.470978    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-volume\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.471046 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.471009    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-web-config\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.471156 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.471064    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-out\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.471156 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.471114    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-metrics-client-ca\") pod \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\" (UID: \"30a086dd-07a4-4ebb-bce8-cbfb0aff030f\") "
Jul 29 20:28:02.471581 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.471385    2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-main-db\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.471773 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.471742    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:28:02.471847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.471756    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:28:02.472989 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.472949    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-kube-api-access-4f4p6" (OuterVolumeSpecName: "kube-api-access-4f4p6") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "kube-api-access-4f4p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:28:02.473272 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.473246    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:02.473945 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.473913    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:02.474138 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.474105    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:02.474215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.474178    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:28:02.474215 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.474148    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:02.474772 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.474747    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-volume" (OuterVolumeSpecName: "config-volume") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:02.474857 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.474756    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-out" (OuterVolumeSpecName: "config-out") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:28:02.477467 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.477438    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:02.483779 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.483759    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-web-config" (OuterVolumeSpecName: "web-config") pod "30a086dd-07a4-4ebb-bce8-cbfb0aff030f" (UID: "30a086dd-07a4-4ebb-bce8-cbfb0aff030f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:02.516947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.516924    2585 generic.go:356] "Generic (PLEG): container finished" podID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerID="0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c" exitCode=0
Jul 29 20:28:02.516947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.516944    2585 generic.go:356] "Generic (PLEG): container finished" podID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerID="e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb" exitCode=0
Jul 29 20:28:02.517099 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.516977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerDied","Data":"0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c"}
Jul 29 20:28:02.517099 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.517018    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.517099 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.517057    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerDied","Data":"e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb"}
Jul 29 20:28:02.517099 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.517084    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30a086dd-07a4-4ebb-bce8-cbfb0aff030f","Type":"ContainerDied","Data":"b0ea1bec0bbf514ec670690f5ca6e6f60cfd2057191cdb7bc78f4965339e22a5"}
Jul 29 20:28:02.517297 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.517113    2585 scope.go:117] "RemoveContainer" containerID="b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465"
Jul 29 20:28:02.525234 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.525214    2585 scope.go:117] "RemoveContainer" containerID="0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c"
Jul 29 20:28:02.531830 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.531812    2585 scope.go:117] "RemoveContainer" containerID="d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814"
Jul 29 20:28:02.537987 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.537971    2585 scope.go:117] "RemoveContainer" containerID="e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb"
Jul 29 20:28:02.542731 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.542710    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 20:28:02.545017 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.545003    2585 scope.go:117] "RemoveContainer" containerID="6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072"
Jul 29 20:28:02.549107 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.549090    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 20:28:02.552379 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.552357    2585 scope.go:117] "RemoveContainer" containerID="239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c"
Jul 29 20:28:02.558799 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.558782    2585 scope.go:117] "RemoveContainer" containerID="d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da"
Jul 29 20:28:02.565021 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.565000    2585 scope.go:117] "RemoveContainer" containerID="b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465"
Jul 29 20:28:02.565613 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:28:02.565584    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465\": container with ID starting with b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465 not found: ID does not exist" containerID="b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465"
Jul 29 20:28:02.565710 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.565625    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465"} err="failed to get container status \"b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465\": rpc error: code = NotFound desc = could not find container \"b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465\": container with ID starting with b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465 not found: ID does not exist"
Jul 29 20:28:02.565710 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.565651    2585 scope.go:117] "RemoveContainer" containerID="0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c"
Jul 29 20:28:02.565998 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:28:02.565978    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c\": container with ID starting with 0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c not found: ID does not exist" containerID="0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c"
Jul 29 20:28:02.566058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.566004    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c"} err="failed to get container status \"0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c\": rpc error: code = NotFound desc = could not find container \"0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c\": container with ID starting with 0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c not found: ID does not exist"
Jul 29 20:28:02.566058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.566019    2585 scope.go:117] "RemoveContainer" containerID="d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814"
Jul 29 20:28:02.566255 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:28:02.566242    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814\": container with ID starting with d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814 not found: ID does not exist" containerID="d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814"
Jul 29 20:28:02.566299 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.566257    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814"} err="failed to get container status \"d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814\": rpc error: code = NotFound desc = could not find container \"d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814\": container with ID starting with d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814 not found: ID does not exist"
Jul 29 20:28:02.566299 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.566268    2585 scope.go:117] "RemoveContainer" containerID="e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb"
Jul 29 20:28:02.566508 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:28:02.566488    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb\": container with ID starting with e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb not found: ID does not exist" containerID="e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb"
Jul 29 20:28:02.566572 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.566516    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb"} err="failed to get container status \"e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb\": rpc error: code = NotFound desc = could not find container \"e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb\": container with ID starting with e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb not found: ID does not exist"
Jul 29 20:28:02.566572 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.566537    2585 scope.go:117] "RemoveContainer" containerID="6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072"
Jul 29 20:28:02.566783 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:28:02.566767    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072\": container with ID starting with 6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072 not found: ID does not exist" containerID="6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072"
Jul 29 20:28:02.566844 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.566790    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072"} err="failed to get container status \"6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072\": rpc error: code = NotFound desc = could not find container \"6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072\": container with ID starting with 6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072 not found: ID does not exist"
Jul 29 20:28:02.566844 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.566810    2585 scope.go:117] "RemoveContainer" containerID="239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c"
Jul 29 20:28:02.567017 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:28:02.567002    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c\": container with ID starting with 239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c not found: ID does not exist" containerID="239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c"
Jul 29 20:28:02.567080 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567023    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c"} err="failed to get container status \"239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c\": rpc error: code = NotFound desc = could not find container \"239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c\": container with ID starting with 239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c not found: ID does not exist"
Jul 29 20:28:02.567080 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567040    2585 scope.go:117] "RemoveContainer" containerID="d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da"
Jul 29 20:28:02.567260 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:28:02.567242    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da\": container with ID starting with d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da not found: ID does not exist" containerID="d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da"
Jul 29 20:28:02.567319 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567266    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da"} err="failed to get container status \"d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da\": rpc error: code = NotFound desc = could not find container \"d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da\": container with ID starting with d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da not found: ID does not exist"
Jul 29 20:28:02.567319 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567284    2585 scope.go:117] "RemoveContainer" containerID="b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465"
Jul 29 20:28:02.567541 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567524    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465"} err="failed to get container status \"b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465\": rpc error: code = NotFound desc = could not find container \"b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465\": container with ID starting with b30cd0399a7adf1603cbfb2afe24e395eabd2baf824660e17b8922207663f465 not found: ID does not exist"
Jul 29 20:28:02.567580 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567542    2585 scope.go:117] "RemoveContainer" containerID="0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c"
Jul 29 20:28:02.567747 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567729    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c"} err="failed to get container status \"0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c\": rpc error: code = NotFound desc = could not find container \"0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c\": container with ID starting with 0e02929420cbbfa36fcab91917a797ea76e6496a283f9dc6b1790cdc51b99b9c not found: ID does not exist"
Jul 29 20:28:02.567797 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567748    2585 scope.go:117] "RemoveContainer" containerID="d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814"
Jul 29 20:28:02.567959 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567943    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814"} err="failed to get container status \"d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814\": rpc error: code = NotFound desc = could not find container \"d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814\": container with ID starting with d35929ce6cb269b30ab10dff35bf5884565f2d6449c47e4e7d504796af42e814 not found: ID does not exist"
Jul 29 20:28:02.568003 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.567960    2585 scope.go:117] "RemoveContainer" containerID="e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb"
Jul 29 20:28:02.568143 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.568128    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb"} err="failed to get container status \"e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb\": rpc error: code = NotFound desc = could not find container \"e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb\": container with ID starting with e2684dd43d06d832d8b9d88a4612a93bf9e92c2ff263717bb6ad851c5ad6adcb not found: ID does not exist"
Jul 29 20:28:02.568182 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.568143    2585 scope.go:117] "RemoveContainer" containerID="6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072"
Jul 29 20:28:02.568320 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.568297    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072"} err="failed to get container status \"6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072\": rpc error: code = NotFound desc = could not find container \"6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072\": container with ID starting with 6387a2eb3b7724200d64cfab730c4740bfd54d9e690791c170e05d0870a79072 not found: ID does not exist"
Jul 29 20:28:02.568320 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.568314    2585 scope.go:117] "RemoveContainer" containerID="239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c"
Jul 29 20:28:02.568525 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.568506    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c"} err="failed to get container status \"239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c\": rpc error: code = NotFound desc = could not find container \"239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c\": container with ID starting with 239895a3935e6f3527e6fd7ee46347ef7100769007de19f1cb6ba34aebfd993c not found: ID does not exist"
Jul 29 20:28:02.568575 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.568528    2585 scope.go:117] "RemoveContainer" containerID="d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da"
Jul 29 20:28:02.568722 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.568705    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da"} err="failed to get container status \"d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da\": rpc error: code = NotFound desc = could not find container \"d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da\": container with ID starting with d5ebd27bea25e7afc92d78d8fb447127966d58373b2916f27c64ab662d6318da not found: ID does not exist"
Jul 29 20:28:02.572095 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572075    2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572097    2585 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572106    2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-web-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572114    2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-config-out\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572122    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-metrics-client-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572131    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572140    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572151    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4p6\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-kube-api-access-4f4p6\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572159    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572167    2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-tls-assets\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572527 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572175    2585 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-cluster-tls-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.572527 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.572183    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30a086dd-07a4-4ebb-bce8-cbfb0aff030f-secret-alertmanager-main-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:02.576055 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576031    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 20:28:02.576514 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576495    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="config-reloader"
Jul 29 20:28:02.576605 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576519    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="config-reloader"
Jul 29 20:28:02.576605 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576553    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="prom-label-proxy"
Jul 29 20:28:02.576605 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576564    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="prom-label-proxy"
Jul 29 20:28:02.576605 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576580    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2883174b-258b-4a7b-906d-237dc51e8027" containerName="registry"
Jul 29 20:28:02.576605 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576589    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2883174b-258b-4a7b-906d-237dc51e8027" containerName="registry"
Jul 29 20:28:02.576605 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576601    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy-web"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576610    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy-web"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576621    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576631    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576649    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="init-config-reloader"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576658    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="init-config-reloader"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576669    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy-metric"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576678    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy-metric"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576695    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="alertmanager"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576703    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="alertmanager"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576775    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="prom-label-proxy"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576787    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="alertmanager"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576798    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy-web"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576808    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy-metric"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576816    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="kube-rbac-proxy"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576827    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" containerName="config-reloader"
Jul 29 20:28:02.576895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.576835    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2883174b-258b-4a7b-906d-237dc51e8027" containerName="registry"
Jul 29 20:28:02.584456 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.584441    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.587375 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.587354    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 20:28:02.587375 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.587369    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 20:28:02.587516 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.587360    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-spqhl\""
Jul 29 20:28:02.587516 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.587505    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 20:28:02.587591 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.587527    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 20:28:02.587665 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.587648    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 20:28:02.587906 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.587891    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 20:28:02.587985 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.587893    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 20:28:02.587985 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.587920    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 20:28:02.592958 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.592940    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 20:28:02.598816 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.598794    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 20:28:02.673255 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673228    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-web-config\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673365 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673264    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673365 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673291    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e493c-ec2d-4a36-8815-75abe62767f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673458 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673356    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6e493c-ec2d-4a36-8815-75abe62767f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673458 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673403    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673535 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673488    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673535 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673508    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673535 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673525    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwq2\" (UniqueName: \"kubernetes.io/projected/4b6e493c-ec2d-4a36-8815-75abe62767f7-kube-api-access-wpwq2\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673621 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673553    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4b6e493c-ec2d-4a36-8815-75abe62767f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673621 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673572    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673621 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673606    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b6e493c-ec2d-4a36-8815-75abe62767f7-config-out\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673711 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673622    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b6e493c-ec2d-4a36-8815-75abe62767f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.673711 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.673639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.774898 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.774835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.774898 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.774870    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775061 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.774902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwq2\" (UniqueName: \"kubernetes.io/projected/4b6e493c-ec2d-4a36-8815-75abe62767f7-kube-api-access-wpwq2\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775061 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.774926    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4b6e493c-ec2d-4a36-8815-75abe62767f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775061 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.774951    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775061 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.775005    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b6e493c-ec2d-4a36-8815-75abe62767f7-config-out\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775061 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.775030    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b6e493c-ec2d-4a36-8815-75abe62767f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775061 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.775055    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.775098    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-web-config\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.775130    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.775168    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e493c-ec2d-4a36-8815-75abe62767f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.775194    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6e493c-ec2d-4a36-8815-75abe62767f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.775230    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.775622 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.775427    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4b6e493c-ec2d-4a36-8815-75abe62767f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778075 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.777979    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778075 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.777979    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778075 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.778006    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b6e493c-ec2d-4a36-8815-75abe62767f7-config-out\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778075 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.778048    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b6e493c-ec2d-4a36-8815-75abe62767f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778075 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.778050    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778417 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.778171    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-web-config\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778417 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.778257    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778417 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.778313    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778720 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.778696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e493c-ec2d-4a36-8815-75abe62767f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.778757 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.778748    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6e493c-ec2d-4a36-8815-75abe62767f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.779763 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.779744    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b6e493c-ec2d-4a36-8815-75abe62767f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.784306 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.784289    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwq2\" (UniqueName: \"kubernetes.io/projected/4b6e493c-ec2d-4a36-8815-75abe62767f7-kube-api-access-wpwq2\") pod \"alertmanager-main-0\" (UID: \"4b6e493c-ec2d-4a36-8815-75abe62767f7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:02.893484 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:02.893460    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 20:28:03.021591 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:03.021565    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 20:28:03.023680 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:28:03.023653    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6e493c_ec2d_4a36_8815_75abe62767f7.slice/crio-78d43f3f9b90a1dc9b365c4bfa2292e0e09fbef473de5e55a28d040ab72ce928 WatchSource:0}: Error finding container 78d43f3f9b90a1dc9b365c4bfa2292e0e09fbef473de5e55a28d040ab72ce928: Status 404 returned error can't find the container with id 78d43f3f9b90a1dc9b365c4bfa2292e0e09fbef473de5e55a28d040ab72ce928
Jul 29 20:28:03.522870 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:03.522838    2585 generic.go:356] "Generic (PLEG): container finished" podID="4b6e493c-ec2d-4a36-8815-75abe62767f7" containerID="0e2ad0e6dafa3f9f65c9d0a72d4c5420625aedef09dc58459829d6dd02485567" exitCode=0
Jul 29 20:28:03.523484 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:03.523458    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a086dd-07a4-4ebb-bce8-cbfb0aff030f" path="/var/lib/kubelet/pods/30a086dd-07a4-4ebb-bce8-cbfb0aff030f/volumes"
Jul 29 20:28:03.524164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:03.524138    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b6e493c-ec2d-4a36-8815-75abe62767f7","Type":"ContainerDied","Data":"0e2ad0e6dafa3f9f65c9d0a72d4c5420625aedef09dc58459829d6dd02485567"}
Jul 29 20:28:03.524244 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:03.524212    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b6e493c-ec2d-4a36-8815-75abe62767f7","Type":"ContainerStarted","Data":"78d43f3f9b90a1dc9b365c4bfa2292e0e09fbef473de5e55a28d040ab72ce928"}
Jul 29 20:28:04.531857 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:04.531825    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b6e493c-ec2d-4a36-8815-75abe62767f7","Type":"ContainerStarted","Data":"934263fcdaca2dc0f407ffaeb4bafa4b6a49f15bc836bf24aaa706947aa9479f"}
Jul 29 20:28:04.532229 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:04.531883    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b6e493c-ec2d-4a36-8815-75abe62767f7","Type":"ContainerStarted","Data":"e9c5990f5f6495b4210b01c283d033ffc6677f66ac1c4b2260377864f67cc8e0"}
Jul 29 20:28:04.532229 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:04.531909    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b6e493c-ec2d-4a36-8815-75abe62767f7","Type":"ContainerStarted","Data":"2e698756eecd98a6451b11dbd1a1bea319e380962e4259d641349dbc68e64b6e"}
Jul 29 20:28:04.532229 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:04.531931    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b6e493c-ec2d-4a36-8815-75abe62767f7","Type":"ContainerStarted","Data":"be4d4c0ff3fcf2686cc196abe277c33d39944361e1d48f6366fca6c12b42e150"}
Jul 29 20:28:04.532229 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:04.531955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b6e493c-ec2d-4a36-8815-75abe62767f7","Type":"ContainerStarted","Data":"99a97daea8168b7de30b08c2cc5c5110de446875c9deba8119579d3787d80385"}
Jul 29 20:28:04.532229 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:04.531978    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b6e493c-ec2d-4a36-8815-75abe62767f7","Type":"ContainerStarted","Data":"68d99b8f5f17973dd78fe039db69aea240cfb91e2d5d22f90c1375459498cd0a"}
Jul 29 20:28:04.558913 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:04.558858    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.54437085 podStartE2EDuration="2.558844499s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.473649ms" isStatefulPod=false podCreationTimestamp="2026-07-29 20:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:28:04.557361012 +0000 UTC m=+211.601053198" watchObservedRunningTime="2026-07-29 20:28:04.558844499 +0000 UTC m=+211.602536683"
Jul 29 20:28:05.328068 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.328032    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 20:28:05.328417 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.328378    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="prometheus" containerID="cri-o://a4871d493ae0c26bd8f3859ec8d5b7a523a1d376868a05cef9991118254ab7bc" gracePeriod=600
Jul 29 20:28:05.328417 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.328396    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy" containerID="cri-o://8748ab872a7d9b59d463b616fdddea6241d268beacb72df7265cb9fb68fdc7c8" gracePeriod=600
Jul 29 20:28:05.328647 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.328428    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b971a0c92d4b2272419b2031a1fe761109058dfa18e957bbb906d4fbe3ebd87a" gracePeriod=600
Jul 29 20:28:05.328647 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.328454    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy-web" containerID="cri-o://b8b3f16041603fc58e37488e6857e2f6a8efc7535ee9f6cc51773f62a3a0e521" gracePeriod=600
Jul 29 20:28:05.328647 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.328431    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="thanos-sidecar" containerID="cri-o://d83cd1e28b751960ee9e5cd0b5acffe37205ea34de4c774b7aaa4d12400c2105" gracePeriod=600
Jul 29 20:28:05.328647 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.328461    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="config-reloader" containerID="cri-o://75223a741b8b3f8799de6ef4fc184a51ef2c175574d962da94920693f61bbac5" gracePeriod=600
Jul 29 20:28:05.539631 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539603    2585 generic.go:356] "Generic (PLEG): container finished" podID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerID="b971a0c92d4b2272419b2031a1fe761109058dfa18e957bbb906d4fbe3ebd87a" exitCode=0
Jul 29 20:28:05.539631 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539627    2585 generic.go:356] "Generic (PLEG): container finished" podID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerID="8748ab872a7d9b59d463b616fdddea6241d268beacb72df7265cb9fb68fdc7c8" exitCode=0
Jul 29 20:28:05.539631 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539634    2585 generic.go:356] "Generic (PLEG): container finished" podID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerID="b8b3f16041603fc58e37488e6857e2f6a8efc7535ee9f6cc51773f62a3a0e521" exitCode=0
Jul 29 20:28:05.540128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539642    2585 generic.go:356] "Generic (PLEG): container finished" podID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerID="d83cd1e28b751960ee9e5cd0b5acffe37205ea34de4c774b7aaa4d12400c2105" exitCode=0
Jul 29 20:28:05.540128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539633    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerDied","Data":"b971a0c92d4b2272419b2031a1fe761109058dfa18e957bbb906d4fbe3ebd87a"}
Jul 29 20:28:05.540128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539697    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerDied","Data":"8748ab872a7d9b59d463b616fdddea6241d268beacb72df7265cb9fb68fdc7c8"}
Jul 29 20:28:05.540128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539648    2585 generic.go:356] "Generic (PLEG): container finished" podID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerID="75223a741b8b3f8799de6ef4fc184a51ef2c175574d962da94920693f61bbac5" exitCode=0
Jul 29 20:28:05.540128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539721    2585 generic.go:356] "Generic (PLEG): container finished" podID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerID="a4871d493ae0c26bd8f3859ec8d5b7a523a1d376868a05cef9991118254ab7bc" exitCode=0
Jul 29 20:28:05.540128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539728    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerDied","Data":"b8b3f16041603fc58e37488e6857e2f6a8efc7535ee9f6cc51773f62a3a0e521"}
Jul 29 20:28:05.540128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539754    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerDied","Data":"d83cd1e28b751960ee9e5cd0b5acffe37205ea34de4c774b7aaa4d12400c2105"}
Jul 29 20:28:05.540128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539780    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerDied","Data":"75223a741b8b3f8799de6ef4fc184a51ef2c175574d962da94920693f61bbac5"}
Jul 29 20:28:05.540128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.539807    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerDied","Data":"a4871d493ae0c26bd8f3859ec8d5b7a523a1d376868a05cef9991118254ab7bc"}
Jul 29 20:28:05.566279 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.566259    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:05.697564 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697492    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-db\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.697564 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697532    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-web-config\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.697733 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697583    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-trusted-ca-bundle\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.697733 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697615    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-thanos-prometheus-http-client-file\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.697733 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697644    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-kubelet-serving-ca-bundle\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.697733 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697673    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-tls\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.697733 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697705    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.697733 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697728    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-grpc-tls\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697757    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-metrics-client-ca\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697782    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-metrics-client-certs\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697822    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697857    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-0\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697900    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-1\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697927    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78jbq\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-kube-api-access-78jbq\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697965    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-config-out\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697989    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-kube-rbac-proxy\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.697994    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:28:05.698015 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.698012    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-tls-assets\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698557 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.698046    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-2\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698557 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.698074    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-config\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698557 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.698078    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:28:05.698557 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.698115    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-serving-certs-ca-bundle\") pod \"66618f3e-62a8-484b-9da3-0f0e404e3667\" (UID: \"66618f3e-62a8-484b-9da3-0f0e404e3667\") "
Jul 29 20:28:05.698557 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.698318    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.698557 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.698351    2585 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.698863 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.698582    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:28:05.698863 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.698640    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:28:05.699589 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.699402    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:28:05.699589 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.699464    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:28:05.699987 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.699961    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:28:05.700110 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.700054    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:28:05.700508 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.700476    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:05.701917 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.701890    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:05.702017 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.701922    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:05.702017 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.701947    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:05.702017 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.701973    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:05.702247 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.702217    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:05.702399 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.702379    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-config-out" (OuterVolumeSpecName: "config-out") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:28:05.702786 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.702750    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:05.702871 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.702765    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:28:05.702871 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.702841    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-kube-api-access-78jbq" (OuterVolumeSpecName: "kube-api-access-78jbq") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "kube-api-access-78jbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:28:05.703082 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.703066    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-config" (OuterVolumeSpecName: "config") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:05.710812 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.710790    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-web-config" (OuterVolumeSpecName: "web-config") pod "66618f3e-62a8-484b-9da3-0f0e404e3667" (UID: "66618f3e-62a8-484b-9da3-0f0e404e3667"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:28:05.798866 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798833    2585 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.798866 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798861    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-db\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.798866 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798871    2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-web-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798880    2585 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798889    2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798900    2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798908    2585 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-grpc-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798917    2585 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-configmap-metrics-client-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798926    2585 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-metrics-client-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798935    2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798944    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798954    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798962    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78jbq\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-kube-api-access-78jbq\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798970    2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66618f3e-62a8-484b-9da3-0f0e404e3667-config-out\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798990    2585 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-secret-kube-rbac-proxy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.798998    2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66618f3e-62a8-484b-9da3-0f0e404e3667-tls-assets\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.799007    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66618f3e-62a8-484b-9da3-0f0e404e3667-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:05.799058 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:05.799015    2585 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/66618f3e-62a8-484b-9da3-0f0e404e3667-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:28:06.547622 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.547586    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66618f3e-62a8-484b-9da3-0f0e404e3667","Type":"ContainerDied","Data":"12569173e0b52bf05a0104d558344885a7dc63b5796c18ac14956cfd8ce65047"}
Jul 29 20:28:06.548010 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.547644    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.548010 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.547656    2585 scope.go:117] "RemoveContainer" containerID="b971a0c92d4b2272419b2031a1fe761109058dfa18e957bbb906d4fbe3ebd87a"
Jul 29 20:28:06.556238 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.556202    2585 scope.go:117] "RemoveContainer" containerID="8748ab872a7d9b59d463b616fdddea6241d268beacb72df7265cb9fb68fdc7c8"
Jul 29 20:28:06.563001 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.562981    2585 scope.go:117] "RemoveContainer" containerID="b8b3f16041603fc58e37488e6857e2f6a8efc7535ee9f6cc51773f62a3a0e521"
Jul 29 20:28:06.569993 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.569975    2585 scope.go:117] "RemoveContainer" containerID="d83cd1e28b751960ee9e5cd0b5acffe37205ea34de4c774b7aaa4d12400c2105"
Jul 29 20:28:06.574173 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.574153    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 20:28:06.577223 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.577205    2585 scope.go:117] "RemoveContainer" containerID="75223a741b8b3f8799de6ef4fc184a51ef2c175574d962da94920693f61bbac5"
Jul 29 20:28:06.582355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.582315    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 20:28:06.584701 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.584682    2585 scope.go:117] "RemoveContainer" containerID="a4871d493ae0c26bd8f3859ec8d5b7a523a1d376868a05cef9991118254ab7bc"
Jul 29 20:28:06.591637 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.591622    2585 scope.go:117] "RemoveContainer" containerID="47274529b74b8cf169e31ee2b0d8d4964cf0e843e0337bee2451d2e260c89de5"
Jul 29 20:28:06.610243 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610222    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 20:28:06.610513 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610501    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="prometheus"
Jul 29 20:28:06.610554 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610516    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="prometheus"
Jul 29 20:28:06.610554 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610530    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy-web"
Jul 29 20:28:06.610554 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610536    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy-web"
Jul 29 20:28:06.610554 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610545    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="config-reloader"
Jul 29 20:28:06.610554 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610550    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="config-reloader"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610559    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="init-config-reloader"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610564    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="init-config-reloader"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610576    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="thanos-sidecar"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610582    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="thanos-sidecar"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610589    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610593    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610599    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy-thanos"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610604    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy-thanos"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610650    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy-thanos"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610658    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="config-reloader"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610664    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="prometheus"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610669    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy-web"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610674    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="thanos-sidecar"
Jul 29 20:28:06.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.610679    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" containerName="kube-rbac-proxy"
Jul 29 20:28:06.615817 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.615801    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.619103 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.619058    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 20:28:06.619103 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.619071    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 20:28:06.619103 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.619081    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 20:28:06.619422 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.619388    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 20:28:06.619965 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.619948    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d7ts5iqpsdi2t\""
Jul 29 20:28:06.619965 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.619957    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 20:28:06.620086 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.620008    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 20:28:06.620543 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.620526    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 20:28:06.620917 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.620891    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 20:28:06.621189 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.621163    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 20:28:06.621285 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.621193    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-prk6z\""
Jul 29 20:28:06.621413 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.621289    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 20:28:06.625059 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.624476    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 20:28:06.625059 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.624510    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 20:28:06.625792 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.625772    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 20:28:06.627770 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.627727    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 20:28:06.630067 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.630045    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 20:28:06.636857 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.636839    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 20:28:06.707360 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707314    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707486 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707367    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-config\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707486 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707391    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707486 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707425    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707486 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707473    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707512    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707534    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707554    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26b8dfa8-6577-40d5-8ad5-197f5d97660d-config-out\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707572    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707738 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707614    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707738 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707640    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26b8dfa8-6577-40d5-8ad5-197f5d97660d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707738 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707738 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707702    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707738 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707732    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707879 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707753    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707879 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707770    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707879 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707819    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-web-config\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707879 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707833    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707879 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707846    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrpp\" (UniqueName: \"kubernetes.io/projected/26b8dfa8-6577-40d5-8ad5-197f5d97660d-kube-api-access-twrpp\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.707879 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.707862    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809099    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809128    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809296 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-web-config\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809296 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809164    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809296 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809186    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrpp\" (UniqueName: \"kubernetes.io/projected/26b8dfa8-6577-40d5-8ad5-197f5d97660d-kube-api-access-twrpp\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809296 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809208    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809296 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809236    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809296 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809272    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-config\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809298    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809320    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809366    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809400    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809424    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809455    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26b8dfa8-6577-40d5-8ad5-197f5d97660d-config-out\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809476    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809503    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809535    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26b8dfa8-6577-40d5-8ad5-197f5d97660d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809559    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.809613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.809596    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.811120 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.810283    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.811120 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.810593    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.811120 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.810830    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.811120 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.810917    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.811409 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.811220    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.811578 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.811555    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.812119 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.812091    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.812221 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.812161    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.812979 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.812583    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-web-config\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.812979 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.812941    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.812979 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.812951    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.813236 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.813211    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.813469 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.813390    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26b8dfa8-6577-40d5-8ad5-197f5d97660d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.813869 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.813845    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26b8dfa8-6577-40d5-8ad5-197f5d97660d-config-out\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.814020 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.813995    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.814947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.814923    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.815157 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.815134    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-config\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.815249 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.815179    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.815291 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.815273    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26b8dfa8-6577-40d5-8ad5-197f5d97660d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.815457 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.815436    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26b8dfa8-6577-40d5-8ad5-197f5d97660d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.818002 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.817982    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrpp\" (UniqueName: \"kubernetes.io/projected/26b8dfa8-6577-40d5-8ad5-197f5d97660d-kube-api-access-twrpp\") pod \"prometheus-k8s-0\" (UID: \"26b8dfa8-6577-40d5-8ad5-197f5d97660d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:06.929437 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:06.929371    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:07.062614 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:07.062581    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 20:28:07.065472 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:28:07.065439    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b8dfa8_6577_40d5_8ad5_197f5d97660d.slice/crio-eef4c2070df4453158987ebc722b1bdd2055973ee6731884ca7f45abbe134b0c WatchSource:0}: Error finding container eef4c2070df4453158987ebc722b1bdd2055973ee6731884ca7f45abbe134b0c: Status 404 returned error can't find the container with id eef4c2070df4453158987ebc722b1bdd2055973ee6731884ca7f45abbe134b0c
Jul 29 20:28:07.523399 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:07.523369    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66618f3e-62a8-484b-9da3-0f0e404e3667" path="/var/lib/kubelet/pods/66618f3e-62a8-484b-9da3-0f0e404e3667/volumes"
Jul 29 20:28:07.553962 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:07.553935    2585 generic.go:356] "Generic (PLEG): container finished" podID="26b8dfa8-6577-40d5-8ad5-197f5d97660d" containerID="93127d8f8aa76912fe95c735e33597d904f0669299d90397ef9abe4a32931039" exitCode=0
Jul 29 20:28:07.554304 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:07.553987    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b8dfa8-6577-40d5-8ad5-197f5d97660d","Type":"ContainerDied","Data":"93127d8f8aa76912fe95c735e33597d904f0669299d90397ef9abe4a32931039"}
Jul 29 20:28:07.554304 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:07.554049    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b8dfa8-6577-40d5-8ad5-197f5d97660d","Type":"ContainerStarted","Data":"eef4c2070df4453158987ebc722b1bdd2055973ee6731884ca7f45abbe134b0c"}
Jul 29 20:28:08.052985 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.052906    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:28:08.124303 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.124276    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:28:08.561595 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.561564    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b8dfa8-6577-40d5-8ad5-197f5d97660d","Type":"ContainerStarted","Data":"451bce55fc520e58ccb6baa345f98bd889022a24bf578228e473a264ac224f94"}
Jul 29 20:28:08.562028 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.561620    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b8dfa8-6577-40d5-8ad5-197f5d97660d","Type":"ContainerStarted","Data":"98e1081bc3a88b76db7332796eb6f02cdcf462d5aa50b1c6cb507493deee3687"}
Jul 29 20:28:08.562028 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.561644    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b8dfa8-6577-40d5-8ad5-197f5d97660d","Type":"ContainerStarted","Data":"0c673ba7aea97ea39536c586938dbc02408ca1ae57bb89b120507ddcb40570a6"}
Jul 29 20:28:08.562028 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.561668    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b8dfa8-6577-40d5-8ad5-197f5d97660d","Type":"ContainerStarted","Data":"ebc6ae44f33b9a6bf7381cdb98afb3e257346591f71bbff6623646e3ba376f07"}
Jul 29 20:28:08.562028 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.561691    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b8dfa8-6577-40d5-8ad5-197f5d97660d","Type":"ContainerStarted","Data":"0ee05c7e0cdd2cf34f768a6d5bd51702cc4db686b007925a516907418297c351"}
Jul 29 20:28:08.562028 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.561714    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b8dfa8-6577-40d5-8ad5-197f5d97660d","Type":"ContainerStarted","Data":"8d73173d2916b82ed41545506894da4761b106bd3ae69409801c4d6562bc424c"}
Jul 29 20:28:08.592722 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.592660    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5750325739999997 podStartE2EDuration="2.592646229s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.613655ms" isStatefulPod=false podCreationTimestamp="2026-07-29 20:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:28:08.589961534 +0000 UTC m=+215.633653716" watchObservedRunningTime="2026-07-29 20:28:08.592646229 +0000 UTC m=+215.636338412"
Jul 29 20:28:08.709017 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.708989    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:28:08.809519 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:08.809489    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:28:11.929445 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:11.929410    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:28:14.092053 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:14.092023    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:28:14.193834 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:14.193795    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:28:59.048361 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:59.048263    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:28:59.145669 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:28:59.145636    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:06.929904 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:06.929871    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:29:06.944872 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:06.944850    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:29:07.807306 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:07.807281    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 20:29:25.855539 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:25.855501    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:26.680604 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:26.680569    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:29.457792 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:29.457754    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:29.557933 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:29.557896    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:30.636519 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:30.636481    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:30.752444 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:30.752408    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:31.657584 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:31.657545    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:31.793644 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:31.793614    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:33.376679 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:33.376649    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:29:33.380533 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:33.380508    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:29:33.387732 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:33.387714    2585 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 20:29:33.945512 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:33.945475    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:34.216175 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:34.216136    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:36.654140 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:36.654101    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:36.752687 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:36.752651    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:38.621756 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:38.621725    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:40.698577 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:40.698538    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:40.760233 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:40.760202    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:41.522346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:41.522302    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:53.816494 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:53.816455    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:29:53.911299 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:29:53.911262    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:00.115915 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:00.115877    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:00.215515 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:00.215480    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:01.786358 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:01.786314    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:06.022665 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.022627    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:06.082018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.081974    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"]
Jul 29 20:30:06.086519 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.086494    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.089696 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.089667    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:30:06.091185 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.091162    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 20:30:06.091430 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.091405    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 20:30:06.091532 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.091461    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 20:30:06.091617 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.091595    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 20:30:06.091705 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.091682    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-khxqm\""
Jul 29 20:30:06.098287 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.098261    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"]
Jul 29 20:30:06.105051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.105023    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfvl\" (UniqueName: \"kubernetes.io/projected/59be7ae2-f437-47c5-9d2f-83d34eb48321-kube-api-access-zpfvl\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.105160 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.105056    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59be7ae2-f437-47c5-9d2f-83d34eb48321-metrics-cert\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.105160 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.105081    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59be7ae2-f437-47c5-9d2f-83d34eb48321-cert\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.105160 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.105113    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/59be7ae2-f437-47c5-9d2f-83d34eb48321-manager-config\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.123553 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.123524    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:06.206241 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.206206    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfvl\" (UniqueName: \"kubernetes.io/projected/59be7ae2-f437-47c5-9d2f-83d34eb48321-kube-api-access-zpfvl\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.206241 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.206246    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59be7ae2-f437-47c5-9d2f-83d34eb48321-metrics-cert\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.206505 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.206268    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59be7ae2-f437-47c5-9d2f-83d34eb48321-cert\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.206505 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.206306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/59be7ae2-f437-47c5-9d2f-83d34eb48321-manager-config\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.207013 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.206989    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/59be7ae2-f437-47c5-9d2f-83d34eb48321-manager-config\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.208702 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.208677    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59be7ae2-f437-47c5-9d2f-83d34eb48321-cert\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.208782 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.208714    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59be7ae2-f437-47c5-9d2f-83d34eb48321-metrics-cert\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.215962 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.215938    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfvl\" (UniqueName: \"kubernetes.io/projected/59be7ae2-f437-47c5-9d2f-83d34eb48321-kube-api-access-zpfvl\") pod \"lws-controller-manager-5c7786b84c-pz88f\" (UID: \"59be7ae2-f437-47c5-9d2f-83d34eb48321\") " pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.397906 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.397811    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:06.521724 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.521701    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"]
Jul 29 20:30:06.524026 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:30:06.523997    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59be7ae2_f437_47c5_9d2f_83d34eb48321.slice/crio-54cf85c8f953dd99932f3dd2796f8beb751638d82ff058d361343149a917fd29 WatchSource:0}: Error finding container 54cf85c8f953dd99932f3dd2796f8beb751638d82ff058d361343149a917fd29: Status 404 returned error can't find the container with id 54cf85c8f953dd99932f3dd2796f8beb751638d82ff058d361343149a917fd29
Jul 29 20:30:06.525976 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:06.525961    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:30:07.034926 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:07.034891    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f" event={"ID":"59be7ae2-f437-47c5-9d2f-83d34eb48321","Type":"ContainerStarted","Data":"54cf85c8f953dd99932f3dd2796f8beb751638d82ff058d361343149a917fd29"}
Jul 29 20:30:10.050026 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:10.049996    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f" event={"ID":"59be7ae2-f437-47c5-9d2f-83d34eb48321","Type":"ContainerStarted","Data":"c6b0711598faaa2d754fd3c9af93cdc82294650cad4a7d5ff23b89cae417175c"}
Jul 29 20:30:10.050468 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:10.050115    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:10.067769 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:10.067727    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f" podStartSLOduration=0.986924417 podStartE2EDuration="4.067713348s" totalImagesPullingTime="3.080788931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:30:10.065956702 +0000 UTC m=+337.109648888" watchObservedRunningTime="2026-07-29 20:30:10.067713348 +0000 UTC m=+337.111405531"
Jul 29 20:30:16.770847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:16.770807    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:16.899563 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:16.899531    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:17.389624 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.389591    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:17.629420 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.629388    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:17.733496 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.733463    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:17.859059 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.859023    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"]
Jul 29 20:30:17.862486 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.862466    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:17.869976 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.869957    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 20:30:17.870082 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.869980    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 20:30:17.870368 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.870349    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 20:30:17.870491 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.870476    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k4bvh\""
Jul 29 20:30:17.870579 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.870563    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 20:30:17.889236 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.889197    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"]
Jul 29 20:30:17.901375 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.901352    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-tmp\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:17.901522 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.901389    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-webhook-cert\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:17.901522 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.901430    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjdr\" (UniqueName: \"kubernetes.io/projected/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-kube-api-access-swjdr\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:17.901522 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.901485    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:17.901522 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:17.901509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-manifests\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.002452 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.002368    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-webhook-cert\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.002452 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.002427    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjdr\" (UniqueName: \"kubernetes.io/projected/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-kube-api-access-swjdr\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.002681 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.002469    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.002681 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.002493    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-manifests\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.002681 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.002511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-tmp\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.002985 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.002951    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-manifests\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.004969 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.004938    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.004969 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.004953    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-tmp\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.005125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.004996    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-webhook-cert\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.012848 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.012825    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjdr\" (UniqueName: \"kubernetes.io/projected/39b62403-cc9e-4d4e-bdd3-ee0479a5509f-kube-api-access-swjdr\") pod \"opendatahub-operator-controller-manager-74f7855d79-sn5d6\" (UID: \"39b62403-cc9e-4d4e-bdd3-ee0479a5509f\") " pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.025415 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.025366    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:18.122059 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.122028    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:18.172691 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.172658    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:18.232043 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.232012    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:18.314644 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.314597    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:18.316238 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.316213    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"]
Jul 29 20:30:18.318433 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:30:18.318403    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b62403_cc9e_4d4e_bdd3_ee0479a5509f.slice/crio-7bcdc3acaa8123a7dd4029d616c65a114869a8ece72dc291c1d5eb143af01125 WatchSource:0}: Error finding container 7bcdc3acaa8123a7dd4029d616c65a114869a8ece72dc291c1d5eb143af01125: Status 404 returned error can't find the container with id 7bcdc3acaa8123a7dd4029d616c65a114869a8ece72dc291c1d5eb143af01125
Jul 29 20:30:18.399669 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.399639    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:18.500344 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:18.500293    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:19.089467 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:19.089424    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6" event={"ID":"39b62403-cc9e-4d4e-bdd3-ee0479a5509f","Type":"ContainerStarted","Data":"7bcdc3acaa8123a7dd4029d616c65a114869a8ece72dc291c1d5eb143af01125"}
Jul 29 20:30:21.056855 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:21.056820    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c7786b84c-pz88f"
Jul 29 20:30:22.106672 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:22.106640    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6" event={"ID":"39b62403-cc9e-4d4e-bdd3-ee0479a5509f","Type":"ContainerStarted","Data":"2aedd3302ea34ae5ca9bf9e302e5424330a59e3422a738cffae339d37bc7ee1e"}
Jul 29 20:30:23.112366 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:23.112275    2585 generic.go:356] "Generic (PLEG): container finished" podID="39b62403-cc9e-4d4e-bdd3-ee0479a5509f" containerID="2aedd3302ea34ae5ca9bf9e302e5424330a59e3422a738cffae339d37bc7ee1e" exitCode=0
Jul 29 20:30:23.112752 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:23.112360    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6" event={"ID":"39b62403-cc9e-4d4e-bdd3-ee0479a5509f","Type":"ContainerDied","Data":"2aedd3302ea34ae5ca9bf9e302e5424330a59e3422a738cffae339d37bc7ee1e"}
Jul 29 20:30:24.117968 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:24.117935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6" event={"ID":"39b62403-cc9e-4d4e-bdd3-ee0479a5509f","Type":"ContainerStarted","Data":"18c3dcfc1318ab64c4181cb4c150776d6f79fd5bac6d6cf6ae915bfc47114422"}
Jul 29 20:30:24.118362 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:24.118008    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:24.144754 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:24.144633    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6" podStartSLOduration=2.477410069 podStartE2EDuration="7.144613174s" totalImagesPullingTime="3.61749178s" totalInitContainerRuntime="1.049711325s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:30:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:30:24.14299751 +0000 UTC m=+351.186689698" watchObservedRunningTime="2026-07-29 20:30:24.144613174 +0000 UTC m=+351.188305358"
Jul 29 20:30:35.125007 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:35.124929    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74f7855d79-sn5d6"
Jul 29 20:30:37.456440 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:37.456402    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:38.102118 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.102079    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"]
Jul 29 20:30:38.130615 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.130586    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"]
Jul 29 20:30:38.130785 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.130679    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.135726 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.135702    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 20:30:38.135877 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.135729    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 20:30:38.135877 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.135700    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 20:30:38.135877 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.135703    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 20:30:38.135877 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.135735    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bbg5t\""
Jul 29 20:30:38.271798 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.271763    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a5ce228-0a71-40de-aae7-03f29ce44897-tmp\") pod \"kube-auth-proxy-6b588c995f-5f7xf\" (UID: \"8a5ce228-0a71-40de-aae7-03f29ce44897\") " pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.271970 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.271839    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2s5h\" (UniqueName: \"kubernetes.io/projected/8a5ce228-0a71-40de-aae7-03f29ce44897-kube-api-access-t2s5h\") pod \"kube-auth-proxy-6b588c995f-5f7xf\" (UID: \"8a5ce228-0a71-40de-aae7-03f29ce44897\") " pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.271970 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.271871    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5ce228-0a71-40de-aae7-03f29ce44897-tls-certs\") pod \"kube-auth-proxy-6b588c995f-5f7xf\" (UID: \"8a5ce228-0a71-40de-aae7-03f29ce44897\") " pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.372632 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.372526    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2s5h\" (UniqueName: \"kubernetes.io/projected/8a5ce228-0a71-40de-aae7-03f29ce44897-kube-api-access-t2s5h\") pod \"kube-auth-proxy-6b588c995f-5f7xf\" (UID: \"8a5ce228-0a71-40de-aae7-03f29ce44897\") " pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.372632 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.372604    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5ce228-0a71-40de-aae7-03f29ce44897-tls-certs\") pod \"kube-auth-proxy-6b588c995f-5f7xf\" (UID: \"8a5ce228-0a71-40de-aae7-03f29ce44897\") " pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.372861 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.372644    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a5ce228-0a71-40de-aae7-03f29ce44897-tmp\") pod \"kube-auth-proxy-6b588c995f-5f7xf\" (UID: \"8a5ce228-0a71-40de-aae7-03f29ce44897\") " pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.375165 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.375123    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a5ce228-0a71-40de-aae7-03f29ce44897-tmp\") pod \"kube-auth-proxy-6b588c995f-5f7xf\" (UID: \"8a5ce228-0a71-40de-aae7-03f29ce44897\") " pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.375303 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.375240    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5ce228-0a71-40de-aae7-03f29ce44897-tls-certs\") pod \"kube-auth-proxy-6b588c995f-5f7xf\" (UID: \"8a5ce228-0a71-40de-aae7-03f29ce44897\") " pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.391452 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.391410    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2s5h\" (UniqueName: \"kubernetes.io/projected/8a5ce228-0a71-40de-aae7-03f29ce44897-kube-api-access-t2s5h\") pod \"kube-auth-proxy-6b588c995f-5f7xf\" (UID: \"8a5ce228-0a71-40de-aae7-03f29ce44897\") " pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.441951 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.441916    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"
Jul 29 20:30:38.639914 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.639888    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf"]
Jul 29 20:30:38.641304 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:30:38.641279    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5ce228_0a71_40de_aae7_03f29ce44897.slice/crio-6ca9ec73f59658e04db759c0c48eb7910cc40acc80557ec82b4c10bcf6dff10a WatchSource:0}: Error finding container 6ca9ec73f59658e04db759c0c48eb7910cc40acc80557ec82b4c10bcf6dff10a: Status 404 returned error can't find the container with id 6ca9ec73f59658e04db759c0c48eb7910cc40acc80557ec82b4c10bcf6dff10a
Jul 29 20:30:38.776123 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:38.776089    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:39.182644 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:39.182603    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf" event={"ID":"8a5ce228-0a71-40de-aae7-03f29ce44897","Type":"ContainerStarted","Data":"6ca9ec73f59658e04db759c0c48eb7910cc40acc80557ec82b4c10bcf6dff10a"}
Jul 29 20:30:39.251394 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:39.251309    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:39.497118 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:39.497081    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:39.588276 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:39.588200    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:39.691945 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:39.691900    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:40.682471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:40.682431    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:41.194458 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:41.194424    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf" event={"ID":"8a5ce228-0a71-40de-aae7-03f29ce44897","Type":"ContainerStarted","Data":"78942bcf20057d693fe97f04219aba063c2cacca1b16fd4479b0cba57178aed1"}
Jul 29 20:30:41.213802 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:41.213747    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b588c995f-5f7xf" podStartSLOduration=0.756711174 podStartE2EDuration="3.21373311s" totalImagesPullingTime="2.457021936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:30:41.211023151 +0000 UTC m=+368.254715346" watchObservedRunningTime="2026-07-29 20:30:41.21373311 +0000 UTC m=+368.257425292"
Jul 29 20:30:45.327475 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:45.327439    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:45.426067 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:45.426032    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:46.124178 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:46.124144    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:46.374843 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:46.374733    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:46.507775 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:46.507741    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:48.825023 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:48.824903    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:48.930567 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:48.930523    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:49.000923 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:49.000887    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:49.134189 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:49.134097    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:49.241447 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:49.241412    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:30:49.406732 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:30:49.406647    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:31:22.690109 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:31:22.690069    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:31:22.790243 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:31:22.790210    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:31:38.874463 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:31:38.874416    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:31:38.939502 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:31:38.939468    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:31:39.054922 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:31:39.054885    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:31:58.487424 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:31:58.487320    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:04.839724 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:04.839683    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:04.891983 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:04.891953    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:06.800837 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:06.800801    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:06.902703 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:06.902670    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:07.733010 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:07.732978    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:11.331425 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:11.331387    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:12.683434 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:12.683390    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:12.786431 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:12.786396    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:14.368816 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:14.368781    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:15.806156 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:15.806120    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:16.130991 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:16.130904    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:16.755521 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:16.755487    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:19.173256 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:19.173216    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:20.141470 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:20.141438    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:21.975682 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:21.975648    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"]
Jul 29 20:32:21.978973 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:21.978954    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:21.980169 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:21.980139    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:21.981699 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:21.981678    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 20:32:21.981787 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:21.981697    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 20:32:21.982832 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:21.982814    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-57bgl\""
Jul 29 20:32:21.988589 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:21.988568    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"]
Jul 29 20:32:22.061496 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:22.061465    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkbjf\" (UniqueName: \"kubernetes.io/projected/9280f4b5-10c4-465a-8279-c9d7f6f5d620-kube-api-access-zkbjf\") pod \"limitador-operator-controller-manager-859d7b78bd-fhhc6\" (UID: \"9280f4b5-10c4-465a-8279-c9d7f6f5d620\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:22.162676 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:22.162642    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkbjf\" (UniqueName: \"kubernetes.io/projected/9280f4b5-10c4-465a-8279-c9d7f6f5d620-kube-api-access-zkbjf\") pod \"limitador-operator-controller-manager-859d7b78bd-fhhc6\" (UID: \"9280f4b5-10c4-465a-8279-c9d7f6f5d620\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:22.176228 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:22.176195    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkbjf\" (UniqueName: \"kubernetes.io/projected/9280f4b5-10c4-465a-8279-c9d7f6f5d620-kube-api-access-zkbjf\") pod \"limitador-operator-controller-manager-859d7b78bd-fhhc6\" (UID: \"9280f4b5-10c4-465a-8279-c9d7f6f5d620\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:22.290214 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:22.290129    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:22.430691 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:22.430662    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"]
Jul 29 20:32:22.433405 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:32:22.433376    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9280f4b5_10c4_465a_8279_c9d7f6f5d620.slice/crio-bc20e58cc15fb12c16bb1af3c0e2d3971bb7b7af243901a88ba5cf0f7c033ab6 WatchSource:0}: Error finding container bc20e58cc15fb12c16bb1af3c0e2d3971bb7b7af243901a88ba5cf0f7c033ab6: Status 404 returned error can't find the container with id bc20e58cc15fb12c16bb1af3c0e2d3971bb7b7af243901a88ba5cf0f7c033ab6
Jul 29 20:32:22.641190 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:22.641110    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6" event={"ID":"9280f4b5-10c4-465a-8279-c9d7f6f5d620","Type":"ContainerStarted","Data":"bc20e58cc15fb12c16bb1af3c0e2d3971bb7b7af243901a88ba5cf0f7c033ab6"}
Jul 29 20:32:23.538430 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:23.538390    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:23.639128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:23.639016    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:23.730935 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:23.730893    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:23.787279 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:23.787243    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:23.892533 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:23.892459    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:24.653280 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:24.653249    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6" event={"ID":"9280f4b5-10c4-465a-8279-c9d7f6f5d620","Type":"ContainerStarted","Data":"5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb"}
Jul 29 20:32:24.653742 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:24.653362    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:24.670074 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:24.670012    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6" podStartSLOduration=1.625400934 podStartE2EDuration="3.669993244s" totalImagesPullingTime="2.04459231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:32:24.66876441 +0000 UTC m=+471.712456593" watchObservedRunningTime="2026-07-29 20:32:24.669993244 +0000 UTC m=+471.713685429"
Jul 29 20:32:27.132397 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:27.132353    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:27.232394 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:27.232322    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:34.271584 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.271555    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"]
Jul 29 20:32:34.272114 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.271738    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" containerName="manager" containerID="cri-o://5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb" gracePeriod=2
Jul 29 20:32:34.273831 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.273803    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:34.289071 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.289044    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"]
Jul 29 20:32:34.301512 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.301490    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"]
Jul 29 20:32:34.301867 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.301853    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" containerName="manager"
Jul 29 20:32:34.301909 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.301869    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" containerName="manager"
Jul 29 20:32:34.301943 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.301916    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" containerName="manager"
Jul 29 20:32:34.304403 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:32:34.304374    2585 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9280f4b5-10c4-465a-8279-c9d7f6f5d620\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-fhhc6\": pods \"limitador-operator-controller-manager-859d7b78bd-fhhc6\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:34.304697 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.304684    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"
Jul 29 20:32:34.306825 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:32:34.306676    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fhhc6\" is forbidden: User \"system:node:ip-10-0-136-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-123.ec2.internal' and this object" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:34.308787 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:32:34.308762    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fhhc6\" is forbidden: User \"system:node:ip-10-0-136-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-123.ec2.internal' and this object" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:34.318697 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.318669    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"]
Jul 29 20:32:34.366666 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.366641    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjdm\" (UniqueName: \"kubernetes.io/projected/2eab4c72-f8a5-45e7-88e6-e92919329e31-kube-api-access-9mjdm\") pod \"limitador-operator-controller-manager-859d7b78bd-5g9ct\" (UID: \"2eab4c72-f8a5-45e7-88e6-e92919329e31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"
Jul 29 20:32:34.467827 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.467796    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjdm\" (UniqueName: \"kubernetes.io/projected/2eab4c72-f8a5-45e7-88e6-e92919329e31-kube-api-access-9mjdm\") pod \"limitador-operator-controller-manager-859d7b78bd-5g9ct\" (UID: \"2eab4c72-f8a5-45e7-88e6-e92919329e31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"
Jul 29 20:32:34.476885 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.476856    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjdm\" (UniqueName: \"kubernetes.io/projected/2eab4c72-f8a5-45e7-88e6-e92919329e31-kube-api-access-9mjdm\") pod \"limitador-operator-controller-manager-859d7b78bd-5g9ct\" (UID: \"2eab4c72-f8a5-45e7-88e6-e92919329e31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"
Jul 29 20:32:34.506272 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.506249    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:34.508603 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:32:34.508579    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fhhc6\" is forbidden: User \"system:node:ip-10-0-136-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-123.ec2.internal' and this object" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:34.568823 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.568776    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkbjf\" (UniqueName: \"kubernetes.io/projected/9280f4b5-10c4-465a-8279-c9d7f6f5d620-kube-api-access-zkbjf\") pod \"9280f4b5-10c4-465a-8279-c9d7f6f5d620\" (UID: \"9280f4b5-10c4-465a-8279-c9d7f6f5d620\") "
Jul 29 20:32:34.570720 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.570693    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9280f4b5-10c4-465a-8279-c9d7f6f5d620-kube-api-access-zkbjf" (OuterVolumeSpecName: "kube-api-access-zkbjf") pod "9280f4b5-10c4-465a-8279-c9d7f6f5d620" (UID: "9280f4b5-10c4-465a-8279-c9d7f6f5d620"). InnerVolumeSpecName "kube-api-access-zkbjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:32:34.649820 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.649782    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"
Jul 29 20:32:34.669627 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.669602    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkbjf\" (UniqueName: \"kubernetes.io/projected/9280f4b5-10c4-465a-8279-c9d7f6f5d620-kube-api-access-zkbjf\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:32:34.699866 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.699836    2585 generic.go:356] "Generic (PLEG): container finished" podID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" containerID="5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb" exitCode=0
Jul 29 20:32:34.699958 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.699895    2585 scope.go:117] "RemoveContainer" containerID="5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb"
Jul 29 20:32:34.699958 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.699907    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:34.704378 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:32:34.704340    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fhhc6\" is forbidden: User \"system:node:ip-10-0-136-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-123.ec2.internal' and this object" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:34.712351 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.712304    2585 scope.go:117] "RemoveContainer" containerID="5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb"
Jul 29 20:32:34.713040 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:32:34.712977    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb\": container with ID starting with 5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb not found: ID does not exist" containerID="5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb"
Jul 29 20:32:34.713040 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.713019    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb"} err="failed to get container status \"5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb\": rpc error: code = NotFound desc = could not find container \"5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb\": container with ID starting with 5f09041600a77b64c10d0759c6ce6e0f2c942a0c9884c6f90a07a9d2b8be41eb not found: ID does not exist"
Jul 29 20:32:34.713730 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:32:34.713702    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fhhc6\" is forbidden: User \"system:node:ip-10-0-136-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-123.ec2.internal' and this object" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fhhc6"
Jul 29 20:32:34.791018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:34.790994    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"]
Jul 29 20:32:34.793152 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:32:34.793124    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eab4c72_f8a5_45e7_88e6_e92919329e31.slice/crio-15a2fd82d6693bcbd27dae74e84acd52b9859f60af54bfd1f8897e132d9e59e9 WatchSource:0}: Error finding container 15a2fd82d6693bcbd27dae74e84acd52b9859f60af54bfd1f8897e132d9e59e9: Status 404 returned error can't find the container with id 15a2fd82d6693bcbd27dae74e84acd52b9859f60af54bfd1f8897e132d9e59e9
Jul 29 20:32:35.523626 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:35.523592    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9280f4b5-10c4-465a-8279-c9d7f6f5d620" path="/var/lib/kubelet/pods/9280f4b5-10c4-465a-8279-c9d7f6f5d620/volumes"
Jul 29 20:32:35.706213 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:35.706177    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct" event={"ID":"2eab4c72-f8a5-45e7-88e6-e92919329e31","Type":"ContainerStarted","Data":"f90ad6aeb596d87fe3fc9d70032ea4990428b9668766fac7a5c6c0f160767b8f"}
Jul 29 20:32:35.706390 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:35.706227    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct" event={"ID":"2eab4c72-f8a5-45e7-88e6-e92919329e31","Type":"ContainerStarted","Data":"15a2fd82d6693bcbd27dae74e84acd52b9859f60af54bfd1f8897e132d9e59e9"}
Jul 29 20:32:35.706390 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:35.706260    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"
Jul 29 20:32:35.729567 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:35.729513    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct" podStartSLOduration=1.729498804 podStartE2EDuration="1.729498804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:32:35.727028591 +0000 UTC m=+482.770720775" watchObservedRunningTime="2026-07-29 20:32:35.729498804 +0000 UTC m=+482.773190986"
Jul 29 20:32:37.144382 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:37.144352    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:37.245694 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:37.245661    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:37.733792 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:37.733752    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:37.834287 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:37.834257    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:41.736931 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:41.736898    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:41.836135 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:41.836101    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:44.525263 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:44.525226    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:44.623997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:44.623961    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:46.712739 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:46.712704    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5g9ct"
Jul 29 20:32:48.131577 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:48.131541    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:48.231714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:48.231678    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:57.343134 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:57.343101    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:32:57.442938 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:32:57.442902    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:00.342184 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:00.342143    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:00.442770 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:00.442735    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:15.062086 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:15.062047    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:15.163036 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:15.162997    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:22.061569 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:22.061526    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:22.114233 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:22.114202    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:27.994742 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:27.994653    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:28.106539 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.106497    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:28.161587 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.161553    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:28.776870 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.776838    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-jzjc7"]
Jul 29 20:33:28.779773 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.779753    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jzjc7"
Jul 29 20:33:28.782585 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.782566    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b4567\""
Jul 29 20:33:28.787064 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.787040    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jzjc7"]
Jul 29 20:33:28.811531 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.811500    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:28.830574 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.830551    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhwx\" (UniqueName: \"kubernetes.io/projected/c5b7a816-0e21-495c-8219-f00e75c326d4-kube-api-access-5lhwx\") pod \"authorino-58cd955675-jzjc7\" (UID: \"c5b7a816-0e21-495c-8219-f00e75c326d4\") " pod="kuadrant-system/authorino-58cd955675-jzjc7"
Jul 29 20:33:28.868132 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.868067    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:28.870444 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.870416    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-2fsrr"]
Jul 29 20:33:28.875193 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.875168    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2fsrr"
Jul 29 20:33:28.877502 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.877478    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2fsrr"]
Jul 29 20:33:28.931506 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.931461    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xklh8\" (UniqueName: \"kubernetes.io/projected/1be6b980-ac07-43b6-8cf5-06c3c76cbec9-kube-api-access-xklh8\") pod \"authorino-75d556bd77-2fsrr\" (UID: \"1be6b980-ac07-43b6-8cf5-06c3c76cbec9\") " pod="kuadrant-system/authorino-75d556bd77-2fsrr"
Jul 29 20:33:28.931680 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.931551    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhwx\" (UniqueName: \"kubernetes.io/projected/c5b7a816-0e21-495c-8219-f00e75c326d4-kube-api-access-5lhwx\") pod \"authorino-58cd955675-jzjc7\" (UID: \"c5b7a816-0e21-495c-8219-f00e75c326d4\") " pod="kuadrant-system/authorino-58cd955675-jzjc7"
Jul 29 20:33:28.939564 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:28.939528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhwx\" (UniqueName: \"kubernetes.io/projected/c5b7a816-0e21-495c-8219-f00e75c326d4-kube-api-access-5lhwx\") pod \"authorino-58cd955675-jzjc7\" (UID: \"c5b7a816-0e21-495c-8219-f00e75c326d4\") " pod="kuadrant-system/authorino-58cd955675-jzjc7"
Jul 29 20:33:29.032165 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:29.032073    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xklh8\" (UniqueName: \"kubernetes.io/projected/1be6b980-ac07-43b6-8cf5-06c3c76cbec9-kube-api-access-xklh8\") pod \"authorino-75d556bd77-2fsrr\" (UID: \"1be6b980-ac07-43b6-8cf5-06c3c76cbec9\") " pod="kuadrant-system/authorino-75d556bd77-2fsrr"
Jul 29 20:33:29.040241 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:29.040215    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xklh8\" (UniqueName: \"kubernetes.io/projected/1be6b980-ac07-43b6-8cf5-06c3c76cbec9-kube-api-access-xklh8\") pod \"authorino-75d556bd77-2fsrr\" (UID: \"1be6b980-ac07-43b6-8cf5-06c3c76cbec9\") " pod="kuadrant-system/authorino-75d556bd77-2fsrr"
Jul 29 20:33:29.090516 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:29.090477    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jzjc7"
Jul 29 20:33:29.185794 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:29.185765    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2fsrr"
Jul 29 20:33:29.220161 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:29.219988    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jzjc7"]
Jul 29 20:33:29.222990 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:33:29.222962    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5b7a816_0e21_495c_8219_f00e75c326d4.slice/crio-3bfc2a6a4b017e1ea4ec9d4d424707b2367a52066f36a267ed8317fdd45575c0 WatchSource:0}: Error finding container 3bfc2a6a4b017e1ea4ec9d4d424707b2367a52066f36a267ed8317fdd45575c0: Status 404 returned error can't find the container with id 3bfc2a6a4b017e1ea4ec9d4d424707b2367a52066f36a267ed8317fdd45575c0
Jul 29 20:33:29.315626 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:29.315601    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2fsrr"]
Jul 29 20:33:29.317618 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:33:29.317583    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1be6b980_ac07_43b6_8cf5_06c3c76cbec9.slice/crio-e156057517a51b04fc3e29e51e6e08770cd9290c648a7b3530ed012d27b71eb6 WatchSource:0}: Error finding container e156057517a51b04fc3e29e51e6e08770cd9290c648a7b3530ed012d27b71eb6: Status 404 returned error can't find the container with id e156057517a51b04fc3e29e51e6e08770cd9290c648a7b3530ed012d27b71eb6
Jul 29 20:33:29.953055 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:29.952971    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2fsrr" event={"ID":"1be6b980-ac07-43b6-8cf5-06c3c76cbec9","Type":"ContainerStarted","Data":"e156057517a51b04fc3e29e51e6e08770cd9290c648a7b3530ed012d27b71eb6"}
Jul 29 20:33:29.956533 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:29.956477    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jzjc7" event={"ID":"c5b7a816-0e21-495c-8219-f00e75c326d4","Type":"ContainerStarted","Data":"3bfc2a6a4b017e1ea4ec9d4d424707b2367a52066f36a267ed8317fdd45575c0"}
Jul 29 20:33:32.982133 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:32.982041    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2fsrr" event={"ID":"1be6b980-ac07-43b6-8cf5-06c3c76cbec9","Type":"ContainerStarted","Data":"d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2"}
Jul 29 20:33:32.983592 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:32.983565    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jzjc7" event={"ID":"c5b7a816-0e21-495c-8219-f00e75c326d4","Type":"ContainerStarted","Data":"de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a"}
Jul 29 20:33:32.996858 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:32.996812    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-2fsrr" podStartSLOduration=1.544508085 podStartE2EDuration="4.996797223s" totalImagesPullingTime="3.452289138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:33:32.995591578 +0000 UTC m=+540.039283762" watchObservedRunningTime="2026-07-29 20:33:32.996797223 +0000 UTC m=+540.040489411"
Jul 29 20:33:33.010427 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:33.010371    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-jzjc7" podStartSLOduration=1.452547512 podStartE2EDuration="5.010351133s" totalImagesPullingTime="3.557803621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:33:33.008347811 +0000 UTC m=+540.052039991" watchObservedRunningTime="2026-07-29 20:33:33.010351133 +0000 UTC m=+540.054043315"
Jul 29 20:33:33.031396 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:33.031265    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jzjc7"]
Jul 29 20:33:34.992455 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:34.992416    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-jzjc7" podUID="c5b7a816-0e21-495c-8219-f00e75c326d4" containerName="authorino" containerID="cri-o://de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a" gracePeriod=30
Jul 29 20:33:35.234313 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:35.234286    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jzjc7"
Jul 29 20:33:35.287192 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:35.287118    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lhwx\" (UniqueName: \"kubernetes.io/projected/c5b7a816-0e21-495c-8219-f00e75c326d4-kube-api-access-5lhwx\") pod \"c5b7a816-0e21-495c-8219-f00e75c326d4\" (UID: \"c5b7a816-0e21-495c-8219-f00e75c326d4\") "
Jul 29 20:33:35.289080 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:35.289052    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b7a816-0e21-495c-8219-f00e75c326d4-kube-api-access-5lhwx" (OuterVolumeSpecName: "kube-api-access-5lhwx") pod "c5b7a816-0e21-495c-8219-f00e75c326d4" (UID: "c5b7a816-0e21-495c-8219-f00e75c326d4"). InnerVolumeSpecName "kube-api-access-5lhwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:33:35.388431 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:35.388389    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lhwx\" (UniqueName: \"kubernetes.io/projected/c5b7a816-0e21-495c-8219-f00e75c326d4-kube-api-access-5lhwx\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:33:35.998276 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:35.998239    2585 generic.go:356] "Generic (PLEG): container finished" podID="c5b7a816-0e21-495c-8219-f00e75c326d4" containerID="de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a" exitCode=0
Jul 29 20:33:35.998276 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:35.998276    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jzjc7"
Jul 29 20:33:35.998739 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:35.998319    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jzjc7" event={"ID":"c5b7a816-0e21-495c-8219-f00e75c326d4","Type":"ContainerDied","Data":"de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a"}
Jul 29 20:33:35.998739 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:35.998383    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jzjc7" event={"ID":"c5b7a816-0e21-495c-8219-f00e75c326d4","Type":"ContainerDied","Data":"3bfc2a6a4b017e1ea4ec9d4d424707b2367a52066f36a267ed8317fdd45575c0"}
Jul 29 20:33:35.998739 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:35.998400    2585 scope.go:117] "RemoveContainer" containerID="de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a"
Jul 29 20:33:36.006308 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:36.006289    2585 scope.go:117] "RemoveContainer" containerID="de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a"
Jul 29 20:33:36.006592 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:33:36.006573    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a\": container with ID starting with de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a not found: ID does not exist" containerID="de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a"
Jul 29 20:33:36.006650 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:36.006599    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a"} err="failed to get container status \"de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a\": rpc error: code = NotFound desc = could not find container \"de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a\": container with ID starting with de45521f3176c182e82906ffa3de402e603efa061845c69d00495c443744096a not found: ID does not exist"
Jul 29 20:33:36.014466 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:36.014442    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jzjc7"]
Jul 29 20:33:36.017835 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:36.017813    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-jzjc7"]
Jul 29 20:33:37.523591 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:37.523559    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5b7a816-0e21-495c-8219-f00e75c326d4" path="/var/lib/kubelet/pods/c5b7a816-0e21-495c-8219-f00e75c326d4/volumes"
Jul 29 20:33:48.241702 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:48.241664    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:48.337796 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:48.337767    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:33:48.680756 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:33:48.680680    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:01.364219 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.364174    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:01.586578 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.586540    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jgkn4"]
Jul 29 20:34:01.586934 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.586906    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b7a816-0e21-495c-8219-f00e75c326d4" containerName="authorino"
Jul 29 20:34:01.586934 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.586925    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b7a816-0e21-495c-8219-f00e75c326d4" containerName="authorino"
Jul 29 20:34:01.587121 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.587006    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b7a816-0e21-495c-8219-f00e75c326d4" containerName="authorino"
Jul 29 20:34:01.590255 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.590235    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jgkn4"
Jul 29 20:34:01.597878 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.597852    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jgkn4"]
Jul 29 20:34:01.705659 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.705559    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbfz\" (UniqueName: \"kubernetes.io/projected/cbcf68cd-92d8-426c-aa55-f187e5cb203a-kube-api-access-rgbfz\") pod \"authorino-5d4b67c446-jgkn4\" (UID: \"cbcf68cd-92d8-426c-aa55-f187e5cb203a\") " pod="kuadrant-system/authorino-5d4b67c446-jgkn4"
Jul 29 20:34:01.806309 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.806270    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbfz\" (UniqueName: \"kubernetes.io/projected/cbcf68cd-92d8-426c-aa55-f187e5cb203a-kube-api-access-rgbfz\") pod \"authorino-5d4b67c446-jgkn4\" (UID: \"cbcf68cd-92d8-426c-aa55-f187e5cb203a\") " pod="kuadrant-system/authorino-5d4b67c446-jgkn4"
Jul 29 20:34:01.810245 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.810217    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jgkn4"]
Jul 29 20:34:01.810477 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:34:01.810460    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rgbfz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-jgkn4" podUID="cbcf68cd-92d8-426c-aa55-f187e5cb203a"
Jul 29 20:34:01.820177 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.820154    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbfz\" (UniqueName: \"kubernetes.io/projected/cbcf68cd-92d8-426c-aa55-f187e5cb203a-kube-api-access-rgbfz\") pod \"authorino-5d4b67c446-jgkn4\" (UID: \"cbcf68cd-92d8-426c-aa55-f187e5cb203a\") " pod="kuadrant-system/authorino-5d4b67c446-jgkn4"
Jul 29 20:34:01.830570 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.830543    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b7445cf67-5jcv6"]
Jul 29 20:34:01.833871 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.833853    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7445cf67-5jcv6"
Jul 29 20:34:01.839899 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.839870    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:01.840102 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.840084    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b7445cf67-5jcv6"]
Jul 29 20:34:01.907270 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:01.907234    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jjwf\" (UniqueName: \"kubernetes.io/projected/3880bc83-91c2-48b4-bcaa-3da24ed06fab-kube-api-access-6jjwf\") pod \"authorino-6b7445cf67-5jcv6\" (UID: \"3880bc83-91c2-48b4-bcaa-3da24ed06fab\") " pod="kuadrant-system/authorino-6b7445cf67-5jcv6"
Jul 29 20:34:02.007865 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.007812    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jjwf\" (UniqueName: \"kubernetes.io/projected/3880bc83-91c2-48b4-bcaa-3da24ed06fab-kube-api-access-6jjwf\") pod \"authorino-6b7445cf67-5jcv6\" (UID: \"3880bc83-91c2-48b4-bcaa-3da24ed06fab\") " pod="kuadrant-system/authorino-6b7445cf67-5jcv6"
Jul 29 20:34:02.015843 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.015814    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jjwf\" (UniqueName: \"kubernetes.io/projected/3880bc83-91c2-48b4-bcaa-3da24ed06fab-kube-api-access-6jjwf\") pod \"authorino-6b7445cf67-5jcv6\" (UID: \"3880bc83-91c2-48b4-bcaa-3da24ed06fab\") " pod="kuadrant-system/authorino-6b7445cf67-5jcv6"
Jul 29 20:34:02.098427 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.098386    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b7445cf67-5jcv6"]
Jul 29 20:34:02.098675 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.098661    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7445cf67-5jcv6"
Jul 29 20:34:02.118480 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.118453    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68f9b67975-hxx9g"]
Jul 29 20:34:02.122249 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.122224    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jgkn4"
Jul 29 20:34:02.122540 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.122523    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:34:02.125097 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.125074    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 20:34:02.130145 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.129851    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jgkn4"
Jul 29 20:34:02.130917 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.130891    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68f9b67975-hxx9g"]
Jul 29 20:34:02.208783 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.208753    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgbfz\" (UniqueName: \"kubernetes.io/projected/cbcf68cd-92d8-426c-aa55-f187e5cb203a-kube-api-access-rgbfz\") pod \"cbcf68cd-92d8-426c-aa55-f187e5cb203a\" (UID: \"cbcf68cd-92d8-426c-aa55-f187e5cb203a\") "
Jul 29 20:34:02.209069 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.209042    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3ee8f47-f75b-458e-a119-471f4a0f5120-tls-cert\") pod \"authorino-68f9b67975-hxx9g\" (UID: \"d3ee8f47-f75b-458e-a119-471f4a0f5120\") " pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:34:02.209146 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.209101    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss5cx\" (UniqueName: \"kubernetes.io/projected/d3ee8f47-f75b-458e-a119-471f4a0f5120-kube-api-access-ss5cx\") pod \"authorino-68f9b67975-hxx9g\" (UID: \"d3ee8f47-f75b-458e-a119-471f4a0f5120\") " pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:34:02.211805 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.211759    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcf68cd-92d8-426c-aa55-f187e5cb203a-kube-api-access-rgbfz" (OuterVolumeSpecName: "kube-api-access-rgbfz") pod "cbcf68cd-92d8-426c-aa55-f187e5cb203a" (UID: "cbcf68cd-92d8-426c-aa55-f187e5cb203a"). InnerVolumeSpecName "kube-api-access-rgbfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:34:02.234890 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.234863    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b7445cf67-5jcv6"]
Jul 29 20:34:02.237023 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:34:02.236998    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3880bc83_91c2_48b4_bcaa_3da24ed06fab.slice/crio-01f58463e1f08ba3025c574d1c14ac3b7a92254eb38edeb1747ac63957614b64 WatchSource:0}: Error finding container 01f58463e1f08ba3025c574d1c14ac3b7a92254eb38edeb1747ac63957614b64: Status 404 returned error can't find the container with id 01f58463e1f08ba3025c574d1c14ac3b7a92254eb38edeb1747ac63957614b64
Jul 29 20:34:02.310350 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.310237    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3ee8f47-f75b-458e-a119-471f4a0f5120-tls-cert\") pod \"authorino-68f9b67975-hxx9g\" (UID: \"d3ee8f47-f75b-458e-a119-471f4a0f5120\") " pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:34:02.310350 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.310293    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss5cx\" (UniqueName: \"kubernetes.io/projected/d3ee8f47-f75b-458e-a119-471f4a0f5120-kube-api-access-ss5cx\") pod \"authorino-68f9b67975-hxx9g\" (UID: \"d3ee8f47-f75b-458e-a119-471f4a0f5120\") " pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:34:02.310543 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.310378    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbfz\" (UniqueName: \"kubernetes.io/projected/cbcf68cd-92d8-426c-aa55-f187e5cb203a-kube-api-access-rgbfz\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:34:02.312643 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.312614    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3ee8f47-f75b-458e-a119-471f4a0f5120-tls-cert\") pod \"authorino-68f9b67975-hxx9g\" (UID: \"d3ee8f47-f75b-458e-a119-471f4a0f5120\") " pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:34:02.317989 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.317965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss5cx\" (UniqueName: \"kubernetes.io/projected/d3ee8f47-f75b-458e-a119-471f4a0f5120-kube-api-access-ss5cx\") pod \"authorino-68f9b67975-hxx9g\" (UID: \"d3ee8f47-f75b-458e-a119-471f4a0f5120\") " pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:34:02.438377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.438326    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:34:02.563129 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:02.563104    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68f9b67975-hxx9g"]
Jul 29 20:34:02.564361 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:34:02.564320    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3ee8f47_f75b_458e_a119_471f4a0f5120.slice/crio-ac7e6be57b1f74202ee6cf6687faccc30a6e4baf2cd226751da7a2323bcd767d WatchSource:0}: Error finding container ac7e6be57b1f74202ee6cf6687faccc30a6e4baf2cd226751da7a2323bcd767d: Status 404 returned error can't find the container with id ac7e6be57b1f74202ee6cf6687faccc30a6e4baf2cd226751da7a2323bcd767d
Jul 29 20:34:03.134459 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.134362    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f9b67975-hxx9g" event={"ID":"d3ee8f47-f75b-458e-a119-471f4a0f5120","Type":"ContainerStarted","Data":"5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483"}
Jul 29 20:34:03.134459 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.134412    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f9b67975-hxx9g" event={"ID":"d3ee8f47-f75b-458e-a119-471f4a0f5120","Type":"ContainerStarted","Data":"ac7e6be57b1f74202ee6cf6687faccc30a6e4baf2cd226751da7a2323bcd767d"}
Jul 29 20:34:03.135820 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.135788    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7445cf67-5jcv6" event={"ID":"3880bc83-91c2-48b4-bcaa-3da24ed06fab","Type":"ContainerStarted","Data":"174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282"}
Jul 29 20:34:03.135820 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.135816    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jgkn4"
Jul 29 20:34:03.136024 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.135824    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7445cf67-5jcv6" event={"ID":"3880bc83-91c2-48b4-bcaa-3da24ed06fab","Type":"ContainerStarted","Data":"01f58463e1f08ba3025c574d1c14ac3b7a92254eb38edeb1747ac63957614b64"}
Jul 29 20:34:03.136024 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.135880    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6b7445cf67-5jcv6" podUID="3880bc83-91c2-48b4-bcaa-3da24ed06fab" containerName="authorino" containerID="cri-o://174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282" gracePeriod=30
Jul 29 20:34:03.150251 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.150211    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68f9b67975-hxx9g" podStartSLOduration=0.842072347 podStartE2EDuration="1.150197311s" totalImagesPullingTime="308.124964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:34:03.14858953 +0000 UTC m=+570.192281714" watchObservedRunningTime="2026-07-29 20:34:03.150197311 +0000 UTC m=+570.193889494"
Jul 29 20:34:03.171242 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.171208    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jgkn4"]
Jul 29 20:34:03.177701 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.177667    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2fsrr"]
Jul 29 20:34:03.177901 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.177875    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-2fsrr" podUID="1be6b980-ac07-43b6-8cf5-06c3c76cbec9" containerName="authorino" containerID="cri-o://d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2" gracePeriod=30
Jul 29 20:34:03.179189 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.179170    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jgkn4"]
Jul 29 20:34:03.193568 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.193516    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b7445cf67-5jcv6" podStartSLOduration=1.747843144 podStartE2EDuration="2.193490037s" totalImagesPullingTime="445.646893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:34:03.190935463 +0000 UTC m=+570.234627646" watchObservedRunningTime="2026-07-29 20:34:03.193490037 +0000 UTC m=+570.237182223"
Jul 29 20:34:03.420226 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.420201    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7445cf67-5jcv6"
Jul 29 20:34:03.430980 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.430959    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2fsrr"
Jul 29 20:34:03.522918 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.522892    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xklh8\" (UniqueName: \"kubernetes.io/projected/1be6b980-ac07-43b6-8cf5-06c3c76cbec9-kube-api-access-xklh8\") pod \"1be6b980-ac07-43b6-8cf5-06c3c76cbec9\" (UID: \"1be6b980-ac07-43b6-8cf5-06c3c76cbec9\") "
Jul 29 20:34:03.523297 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.522934    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jjwf\" (UniqueName: \"kubernetes.io/projected/3880bc83-91c2-48b4-bcaa-3da24ed06fab-kube-api-access-6jjwf\") pod \"3880bc83-91c2-48b4-bcaa-3da24ed06fab\" (UID: \"3880bc83-91c2-48b4-bcaa-3da24ed06fab\") "
Jul 29 20:34:03.524892 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.524866    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbcf68cd-92d8-426c-aa55-f187e5cb203a" path="/var/lib/kubelet/pods/cbcf68cd-92d8-426c-aa55-f187e5cb203a/volumes"
Jul 29 20:34:03.525043 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.525021    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3880bc83-91c2-48b4-bcaa-3da24ed06fab-kube-api-access-6jjwf" (OuterVolumeSpecName: "kube-api-access-6jjwf") pod "3880bc83-91c2-48b4-bcaa-3da24ed06fab" (UID: "3880bc83-91c2-48b4-bcaa-3da24ed06fab"). InnerVolumeSpecName "kube-api-access-6jjwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:34:03.525106 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.525025    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be6b980-ac07-43b6-8cf5-06c3c76cbec9-kube-api-access-xklh8" (OuterVolumeSpecName: "kube-api-access-xklh8") pod "1be6b980-ac07-43b6-8cf5-06c3c76cbec9" (UID: "1be6b980-ac07-43b6-8cf5-06c3c76cbec9"). InnerVolumeSpecName "kube-api-access-xklh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:34:03.624026 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.623991    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xklh8\" (UniqueName: \"kubernetes.io/projected/1be6b980-ac07-43b6-8cf5-06c3c76cbec9-kube-api-access-xklh8\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:34:03.624026 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:03.624021    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jjwf\" (UniqueName: \"kubernetes.io/projected/3880bc83-91c2-48b4-bcaa-3da24ed06fab-kube-api-access-6jjwf\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:34:04.141263 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.141172    2585 generic.go:356] "Generic (PLEG): container finished" podID="1be6b980-ac07-43b6-8cf5-06c3c76cbec9" containerID="d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2" exitCode=0
Jul 29 20:34:04.141263 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.141221    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2fsrr"
Jul 29 20:34:04.141520 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.141259    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2fsrr" event={"ID":"1be6b980-ac07-43b6-8cf5-06c3c76cbec9","Type":"ContainerDied","Data":"d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2"}
Jul 29 20:34:04.141520 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.141305    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2fsrr" event={"ID":"1be6b980-ac07-43b6-8cf5-06c3c76cbec9","Type":"ContainerDied","Data":"e156057517a51b04fc3e29e51e6e08770cd9290c648a7b3530ed012d27b71eb6"}
Jul 29 20:34:04.141520 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.141346    2585 scope.go:117] "RemoveContainer" containerID="d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2"
Jul 29 20:34:04.142607 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.142523    2585 generic.go:356] "Generic (PLEG): container finished" podID="3880bc83-91c2-48b4-bcaa-3da24ed06fab" containerID="174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282" exitCode=0
Jul 29 20:34:04.142607 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.142582    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7445cf67-5jcv6"
Jul 29 20:34:04.142720 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.142607    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7445cf67-5jcv6" event={"ID":"3880bc83-91c2-48b4-bcaa-3da24ed06fab","Type":"ContainerDied","Data":"174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282"}
Jul 29 20:34:04.142720 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.142654    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7445cf67-5jcv6" event={"ID":"3880bc83-91c2-48b4-bcaa-3da24ed06fab","Type":"ContainerDied","Data":"01f58463e1f08ba3025c574d1c14ac3b7a92254eb38edeb1747ac63957614b64"}
Jul 29 20:34:04.150051 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.150036    2585 scope.go:117] "RemoveContainer" containerID="d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2"
Jul 29 20:34:04.150279 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:34:04.150261    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2\": container with ID starting with d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2 not found: ID does not exist" containerID="d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2"
Jul 29 20:34:04.150322 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.150286    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2"} err="failed to get container status \"d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2\": rpc error: code = NotFound desc = could not find container \"d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2\": container with ID starting with d9ff5c46ada4ad0c44469d0528a90584ac7c64ce6945e258fc65fe08a55e0af2 not found: ID does not exist"
Jul 29 20:34:04.150322 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.150300    2585 scope.go:117] "RemoveContainer" containerID="174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282"
Jul 29 20:34:04.158237 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.158221    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b7445cf67-5jcv6"]
Jul 29 20:34:04.158295 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.158257    2585 scope.go:117] "RemoveContainer" containerID="174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282"
Jul 29 20:34:04.158522 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:34:04.158506    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282\": container with ID starting with 174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282 not found: ID does not exist" containerID="174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282"
Jul 29 20:34:04.158589 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.158526    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282"} err="failed to get container status \"174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282\": rpc error: code = NotFound desc = could not find container \"174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282\": container with ID starting with 174da4f137057d8f9c3cd419ab4cbb43e2d2b61d1f425b69a2e15f37e9065282 not found: ID does not exist"
Jul 29 20:34:04.161128 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.161109    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b7445cf67-5jcv6"]
Jul 29 20:34:04.171180 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.171159    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2fsrr"]
Jul 29 20:34:04.172912 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:04.172892    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2fsrr"]
Jul 29 20:34:05.524288 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:05.524260    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1be6b980-ac07-43b6-8cf5-06c3c76cbec9" path="/var/lib/kubelet/pods/1be6b980-ac07-43b6-8cf5-06c3c76cbec9/volumes"
Jul 29 20:34:05.524658 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:05.524647    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3880bc83-91c2-48b4-bcaa-3da24ed06fab" path="/var/lib/kubelet/pods/3880bc83-91c2-48b4-bcaa-3da24ed06fab/volumes"
Jul 29 20:34:07.655807 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.655768    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:07.667925 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.667895    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8c47f457-lqwt7"]
Jul 29 20:34:07.668243 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.668231    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3880bc83-91c2-48b4-bcaa-3da24ed06fab" containerName="authorino"
Jul 29 20:34:07.668292 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.668246    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3880bc83-91c2-48b4-bcaa-3da24ed06fab" containerName="authorino"
Jul 29 20:34:07.668292 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.668268    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1be6b980-ac07-43b6-8cf5-06c3c76cbec9" containerName="authorino"
Jul 29 20:34:07.668292 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.668274    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1be6b980-ac07-43b6-8cf5-06c3c76cbec9" containerName="authorino"
Jul 29 20:34:07.668411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.668342    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1be6b980-ac07-43b6-8cf5-06c3c76cbec9" containerName="authorino"
Jul 29 20:34:07.668411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.668355    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3880bc83-91c2-48b4-bcaa-3da24ed06fab" containerName="authorino"
Jul 29 20:34:07.673876 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.673853    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:07.676925 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.676905    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 20:34:07.677059 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.676945    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 20:34:07.677131 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.677067    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ch4nm\""
Jul 29 20:34:07.681562 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.681534    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8c47f457-lqwt7"]
Jul 29 20:34:07.758364 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.758313    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9b878-1c05-4894-97c5-67754da6c8a1-webhook-cert\") pod \"maas-controller-c8c47f457-lqwt7\" (UID: \"a5d9b878-1c05-4894-97c5-67754da6c8a1\") " pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:07.758555 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.758403    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s78r\" (UniqueName: \"kubernetes.io/projected/a5d9b878-1c05-4894-97c5-67754da6c8a1-kube-api-access-7s78r\") pod \"maas-controller-c8c47f457-lqwt7\" (UID: \"a5d9b878-1c05-4894-97c5-67754da6c8a1\") " pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:07.758967 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.758929    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:07.824560 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.824528    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5458c6456b-pvxm6"]
Jul 29 20:34:07.827326 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.827305    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:07.829282 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.829255    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:07.836243 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.836217    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5458c6456b-pvxm6"]
Jul 29 20:34:07.859424 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.859389    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9b878-1c05-4894-97c5-67754da6c8a1-webhook-cert\") pod \"maas-controller-c8c47f457-lqwt7\" (UID: \"a5d9b878-1c05-4894-97c5-67754da6c8a1\") " pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:07.859593 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.859440    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s78r\" (UniqueName: \"kubernetes.io/projected/a5d9b878-1c05-4894-97c5-67754da6c8a1-kube-api-access-7s78r\") pod \"maas-controller-c8c47f457-lqwt7\" (UID: \"a5d9b878-1c05-4894-97c5-67754da6c8a1\") " pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:07.859593 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:34:07.859539    2585 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 29 20:34:07.859690 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:34:07.859600    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5d9b878-1c05-4894-97c5-67754da6c8a1-webhook-cert podName:a5d9b878-1c05-4894-97c5-67754da6c8a1 nodeName:}" failed. No retries permitted until 2026-07-29 20:34:08.359582566 +0000 UTC m=+575.403274736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/a5d9b878-1c05-4894-97c5-67754da6c8a1-webhook-cert") pod "maas-controller-c8c47f457-lqwt7" (UID: "a5d9b878-1c05-4894-97c5-67754da6c8a1") : secret "maas-controller-webhook-cert" not found
Jul 29 20:34:07.867964 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.867933    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s78r\" (UniqueName: \"kubernetes.io/projected/a5d9b878-1c05-4894-97c5-67754da6c8a1-kube-api-access-7s78r\") pod \"maas-controller-c8c47f457-lqwt7\" (UID: \"a5d9b878-1c05-4894-97c5-67754da6c8a1\") " pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:07.960903 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.960811    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55415b07-f850-4ba7-a5dd-e45780a83e6c-webhook-cert\") pod \"maas-controller-5458c6456b-pvxm6\" (UID: \"55415b07-f850-4ba7-a5dd-e45780a83e6c\") " pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:07.961075 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:07.960931    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs96z\" (UniqueName: \"kubernetes.io/projected/55415b07-f850-4ba7-a5dd-e45780a83e6c-kube-api-access-hs96z\") pod \"maas-controller-5458c6456b-pvxm6\" (UID: \"55415b07-f850-4ba7-a5dd-e45780a83e6c\") " pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:08.058256 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.058218    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:08.062278 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.062253    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs96z\" (UniqueName: \"kubernetes.io/projected/55415b07-f850-4ba7-a5dd-e45780a83e6c-kube-api-access-hs96z\") pod \"maas-controller-5458c6456b-pvxm6\" (UID: \"55415b07-f850-4ba7-a5dd-e45780a83e6c\") " pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:08.062425 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.062314    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55415b07-f850-4ba7-a5dd-e45780a83e6c-webhook-cert\") pod \"maas-controller-5458c6456b-pvxm6\" (UID: \"55415b07-f850-4ba7-a5dd-e45780a83e6c\") " pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:08.064648 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.064628    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55415b07-f850-4ba7-a5dd-e45780a83e6c-webhook-cert\") pod \"maas-controller-5458c6456b-pvxm6\" (UID: \"55415b07-f850-4ba7-a5dd-e45780a83e6c\") " pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:08.070024 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.070003    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs96z\" (UniqueName: \"kubernetes.io/projected/55415b07-f850-4ba7-a5dd-e45780a83e6c-kube-api-access-hs96z\") pod \"maas-controller-5458c6456b-pvxm6\" (UID: \"55415b07-f850-4ba7-a5dd-e45780a83e6c\") " pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:08.140067 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.140035    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:08.265283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.265258    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5458c6456b-pvxm6"]
Jul 29 20:34:08.266942 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:34:08.266913    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55415b07_f850_4ba7_a5dd_e45780a83e6c.slice/crio-fb9f1096a771af0df1513c62783d1104c880ce810d67756e439b61169d53694b WatchSource:0}: Error finding container fb9f1096a771af0df1513c62783d1104c880ce810d67756e439b61169d53694b: Status 404 returned error can't find the container with id fb9f1096a771af0df1513c62783d1104c880ce810d67756e439b61169d53694b
Jul 29 20:34:08.365179 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.365140    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9b878-1c05-4894-97c5-67754da6c8a1-webhook-cert\") pod \"maas-controller-c8c47f457-lqwt7\" (UID: \"a5d9b878-1c05-4894-97c5-67754da6c8a1\") " pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:08.367501 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.367474    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9b878-1c05-4894-97c5-67754da6c8a1-webhook-cert\") pod \"maas-controller-c8c47f457-lqwt7\" (UID: \"a5d9b878-1c05-4894-97c5-67754da6c8a1\") " pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:08.585984 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.585903    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:08.714621 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:08.714598    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8c47f457-lqwt7"]
Jul 29 20:34:08.716572 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:34:08.716539    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d9b878_1c05_4894_97c5_67754da6c8a1.slice/crio-d5c3944715f5ff514bdef65fe4c5eb4fc6af7b22351e678f8e94ce9bc0946d87 WatchSource:0}: Error finding container d5c3944715f5ff514bdef65fe4c5eb4fc6af7b22351e678f8e94ce9bc0946d87: Status 404 returned error can't find the container with id d5c3944715f5ff514bdef65fe4c5eb4fc6af7b22351e678f8e94ce9bc0946d87
Jul 29 20:34:09.182607 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:09.182563    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8c47f457-lqwt7" event={"ID":"a5d9b878-1c05-4894-97c5-67754da6c8a1","Type":"ContainerStarted","Data":"d5c3944715f5ff514bdef65fe4c5eb4fc6af7b22351e678f8e94ce9bc0946d87"}
Jul 29 20:34:09.185180 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:09.185145    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-pvxm6" event={"ID":"55415b07-f850-4ba7-a5dd-e45780a83e6c","Type":"ContainerStarted","Data":"fb9f1096a771af0df1513c62783d1104c880ce810d67756e439b61169d53694b"}
Jul 29 20:34:11.074048 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:11.074010    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:11.177926 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:11.177836    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:11.197727 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:11.197691    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8c47f457-lqwt7" event={"ID":"a5d9b878-1c05-4894-97c5-67754da6c8a1","Type":"ContainerStarted","Data":"d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7"}
Jul 29 20:34:11.197866 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:11.197805    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:11.199613 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:11.199585    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-pvxm6" event={"ID":"55415b07-f850-4ba7-a5dd-e45780a83e6c","Type":"ContainerStarted","Data":"d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049"}
Jul 29 20:34:11.199751 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:11.199724    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:11.215828 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:11.215783    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c8c47f457-lqwt7" podStartSLOduration=2.013845526 podStartE2EDuration="4.215767338s" totalImagesPullingTime="2.201921812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:34:11.214138724 +0000 UTC m=+578.257830911" watchObservedRunningTime="2026-07-29 20:34:11.215767338 +0000 UTC m=+578.259459531"
Jul 29 20:34:11.230130 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:11.230082    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5458c6456b-pvxm6" podStartSLOduration=1.5877509189999999 podStartE2EDuration="4.23006806s" totalImagesPullingTime="2.642317141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:34:11.229506512 +0000 UTC m=+578.273198695" watchObservedRunningTime="2026-07-29 20:34:11.23006806 +0000 UTC m=+578.273760243"
Jul 29 20:34:22.211586 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.211557    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:22.212062 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.211633    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:34:22.265263 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.265234    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8c47f457-lqwt7"]
Jul 29 20:34:22.265500 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.265453    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c8c47f457-lqwt7" podUID="a5d9b878-1c05-4894-97c5-67754da6c8a1" containerName="manager" containerID="cri-o://d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7" gracePeriod=30
Jul 29 20:34:22.512183 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.512155    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:22.584955 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.584927    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s78r\" (UniqueName: \"kubernetes.io/projected/a5d9b878-1c05-4894-97c5-67754da6c8a1-kube-api-access-7s78r\") pod \"a5d9b878-1c05-4894-97c5-67754da6c8a1\" (UID: \"a5d9b878-1c05-4894-97c5-67754da6c8a1\") "
Jul 29 20:34:22.585110 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.584970    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9b878-1c05-4894-97c5-67754da6c8a1-webhook-cert\") pod \"a5d9b878-1c05-4894-97c5-67754da6c8a1\" (UID: \"a5d9b878-1c05-4894-97c5-67754da6c8a1\") "
Jul 29 20:34:22.587161 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.587132    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d9b878-1c05-4894-97c5-67754da6c8a1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a5d9b878-1c05-4894-97c5-67754da6c8a1" (UID: "a5d9b878-1c05-4894-97c5-67754da6c8a1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:34:22.587284 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.587195    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d9b878-1c05-4894-97c5-67754da6c8a1-kube-api-access-7s78r" (OuterVolumeSpecName: "kube-api-access-7s78r") pod "a5d9b878-1c05-4894-97c5-67754da6c8a1" (UID: "a5d9b878-1c05-4894-97c5-67754da6c8a1"). InnerVolumeSpecName "kube-api-access-7s78r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:34:22.685582 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.685550    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s78r\" (UniqueName: \"kubernetes.io/projected/a5d9b878-1c05-4894-97c5-67754da6c8a1-kube-api-access-7s78r\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:34:22.685582 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:22.685577    2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9b878-1c05-4894-97c5-67754da6c8a1-webhook-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:34:23.261411 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.261372    2585 generic.go:356] "Generic (PLEG): container finished" podID="a5d9b878-1c05-4894-97c5-67754da6c8a1" containerID="d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7" exitCode=0
Jul 29 20:34:23.261859 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.261420    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8c47f457-lqwt7"
Jul 29 20:34:23.261859 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.261447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8c47f457-lqwt7" event={"ID":"a5d9b878-1c05-4894-97c5-67754da6c8a1","Type":"ContainerDied","Data":"d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7"}
Jul 29 20:34:23.261859 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.261498    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8c47f457-lqwt7" event={"ID":"a5d9b878-1c05-4894-97c5-67754da6c8a1","Type":"ContainerDied","Data":"d5c3944715f5ff514bdef65fe4c5eb4fc6af7b22351e678f8e94ce9bc0946d87"}
Jul 29 20:34:23.261859 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.261527    2585 scope.go:117] "RemoveContainer" containerID="d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7"
Jul 29 20:34:23.270998 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.270824    2585 scope.go:117] "RemoveContainer" containerID="d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7"
Jul 29 20:34:23.271115 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:34:23.271096    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7\": container with ID starting with d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7 not found: ID does not exist" containerID="d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7"
Jul 29 20:34:23.271155 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.271129    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7"} err="failed to get container status \"d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7\": rpc error: code = NotFound desc = could not find container \"d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7\": container with ID starting with d21310f76aa6a0e6db77ae21b2701a77742532d6e677f8f9840a300966ce12d7 not found: ID does not exist"
Jul 29 20:34:23.282319 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.282298    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8c47f457-lqwt7"]
Jul 29 20:34:23.287517 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.287497    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c8c47f457-lqwt7"]
Jul 29 20:34:23.523930 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:23.523844    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d9b878-1c05-4894-97c5-67754da6c8a1" path="/var/lib/kubelet/pods/a5d9b878-1c05-4894-97c5-67754da6c8a1/volumes"
Jul 29 20:34:33.411455 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:33.411423    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:34:33.411909 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:33.411886    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:34:43.635210 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:43.635168    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:43.732347 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:43.732300    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:44.398800 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.398763    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:44.466511 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.466485    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"]
Jul 29 20:34:44.466839 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.466823    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d9b878-1c05-4894-97c5-67754da6c8a1" containerName="manager"
Jul 29 20:34:44.466839 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.466837    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d9b878-1c05-4894-97c5-67754da6c8a1" containerName="manager"
Jul 29 20:34:44.466941 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.466925    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d9b878-1c05-4894-97c5-67754da6c8a1" containerName="manager"
Jul 29 20:34:44.468876 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.468857    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.471730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.471708    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 29 20:34:44.472979 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.472963    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 20:34:44.473172 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.473158    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mt27z\""
Jul 29 20:34:44.473247 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.473207    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 20:34:44.480724 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.480702    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"]
Jul 29 20:34:44.499389 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.499355    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:44.554705 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.554681    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12064095-bb7a-4e22-93b8-a22c90eacd4f-openshift-service-ca\") pod \"maas-api-74b678b674-2wcwv\" (UID: \"12064095-bb7a-4e22-93b8-a22c90eacd4f\") " pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.554803 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.554714    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12064095-bb7a-4e22-93b8-a22c90eacd4f-maas-api-tls\") pod \"maas-api-74b678b674-2wcwv\" (UID: \"12064095-bb7a-4e22-93b8-a22c90eacd4f\") " pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.554858 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.554838    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwf2\" (UniqueName: \"kubernetes.io/projected/12064095-bb7a-4e22-93b8-a22c90eacd4f-kube-api-access-gdwf2\") pod \"maas-api-74b678b674-2wcwv\" (UID: \"12064095-bb7a-4e22-93b8-a22c90eacd4f\") " pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.656100 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.656020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12064095-bb7a-4e22-93b8-a22c90eacd4f-openshift-service-ca\") pod \"maas-api-74b678b674-2wcwv\" (UID: \"12064095-bb7a-4e22-93b8-a22c90eacd4f\") " pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.656100 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.656064    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12064095-bb7a-4e22-93b8-a22c90eacd4f-maas-api-tls\") pod \"maas-api-74b678b674-2wcwv\" (UID: \"12064095-bb7a-4e22-93b8-a22c90eacd4f\") " pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.656493 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.656139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwf2\" (UniqueName: \"kubernetes.io/projected/12064095-bb7a-4e22-93b8-a22c90eacd4f-kube-api-access-gdwf2\") pod \"maas-api-74b678b674-2wcwv\" (UID: \"12064095-bb7a-4e22-93b8-a22c90eacd4f\") " pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.656769 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.656749    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12064095-bb7a-4e22-93b8-a22c90eacd4f-openshift-service-ca\") pod \"maas-api-74b678b674-2wcwv\" (UID: \"12064095-bb7a-4e22-93b8-a22c90eacd4f\") " pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.658506 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.658488    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12064095-bb7a-4e22-93b8-a22c90eacd4f-maas-api-tls\") pod \"maas-api-74b678b674-2wcwv\" (UID: \"12064095-bb7a-4e22-93b8-a22c90eacd4f\") " pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.668036 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.668000    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwf2\" (UniqueName: \"kubernetes.io/projected/12064095-bb7a-4e22-93b8-a22c90eacd4f-kube-api-access-gdwf2\") pod \"maas-api-74b678b674-2wcwv\" (UID: \"12064095-bb7a-4e22-93b8-a22c90eacd4f\") " pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.779482 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.779453    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:44.890080 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.890045    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:44.914604 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.914508    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"]
Jul 29 20:34:44.918964 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:34:44.918936    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12064095_bb7a_4e22_93b8_a22c90eacd4f.slice/crio-d85efce437bd7a4e5a8c2bb61e1b4f73f0abcf51f2e5be001e05eec3d50790a2 WatchSource:0}: Error finding container d85efce437bd7a4e5a8c2bb61e1b4f73f0abcf51f2e5be001e05eec3d50790a2: Status 404 returned error can't find the container with id d85efce437bd7a4e5a8c2bb61e1b4f73f0abcf51f2e5be001e05eec3d50790a2
Jul 29 20:34:44.988715 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:44.988683    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:45.365039 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:45.364998    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv" event={"ID":"12064095-bb7a-4e22-93b8-a22c90eacd4f","Type":"ContainerStarted","Data":"d85efce437bd7a4e5a8c2bb61e1b4f73f0abcf51f2e5be001e05eec3d50790a2"}
Jul 29 20:34:47.379710 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:47.379671    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv" event={"ID":"12064095-bb7a-4e22-93b8-a22c90eacd4f","Type":"ContainerStarted","Data":"6934cd279628088cd7ea3f651f615dbd6ab6fb593434c4895aaeae8c74dfed37"}
Jul 29 20:34:47.380173 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:47.379810    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:47.397007 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:47.396964    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv" podStartSLOduration=1.472481387 podStartE2EDuration="3.396948851s" totalImagesPullingTime="1.924467464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:34:47.394805205 +0000 UTC m=+614.438497390" watchObservedRunningTime="2026-07-29 20:34:47.396948851 +0000 UTC m=+614.440641033"
Jul 29 20:34:53.389870 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:53.389844    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74b678b674-2wcwv"
Jul 29 20:34:54.353348 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:54.353292    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:54.425601 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:54.425566    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:34:58.941483 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:34:58.941400    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:35:04.310909 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.310877    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"]
Jul 29 20:35:04.313402 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.313382    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.315928 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.315904    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 20:35:04.315928 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.315915    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bcvdz\""
Jul 29 20:35:04.317202 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.317183    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 29 20:35:04.317318 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.317256    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 20:35:04.324998 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.324976    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"]
Jul 29 20:35:04.341170 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.341142    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:35:04.437438 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.437408    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.437630 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.437462    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944xx\" (UniqueName: \"kubernetes.io/projected/df5bc3cf-a925-49e8-b0ad-148c62e728d3-kube-api-access-944xx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.437630 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.437516    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.437630 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.437568    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.437630 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.437589    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5bc3cf-a925-49e8-b0ad-148c62e728d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.437823 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.437679    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.538462 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.538430    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944xx\" (UniqueName: \"kubernetes.io/projected/df5bc3cf-a925-49e8-b0ad-148c62e728d3-kube-api-access-944xx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.538640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.538469    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.538640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.538498    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.538640 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.538614    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5bc3cf-a925-49e8-b0ad-148c62e728d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.538819 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.538756    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.538819 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.538797    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.538962 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.538940    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.539154 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.539131    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.539200 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.539141    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.540767 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.540745    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5bc3cf-a925-49e8-b0ad-148c62e728d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.541053 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.541033    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5bc3cf-a925-49e8-b0ad-148c62e728d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.545594 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.545572    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-944xx\" (UniqueName: \"kubernetes.io/projected/df5bc3cf-a925-49e8-b0ad-148c62e728d3-kube-api-access-944xx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s\" (UID: \"df5bc3cf-a925-49e8-b0ad-148c62e728d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.625369 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.625277    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:04.760696 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:04.760558    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"]
Jul 29 20:35:04.763302 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:35:04.763273    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5bc3cf_a925_49e8_b0ad_148c62e728d3.slice/crio-2703f000780ee70b63c1f05502277609b09d73049135e5366c6a0655c8b2aacf WatchSource:0}: Error finding container 2703f000780ee70b63c1f05502277609b09d73049135e5366c6a0655c8b2aacf: Status 404 returned error can't find the container with id 2703f000780ee70b63c1f05502277609b09d73049135e5366c6a0655c8b2aacf
Jul 29 20:35:05.465375 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:05.465298    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s" event={"ID":"df5bc3cf-a925-49e8-b0ad-148c62e728d3","Type":"ContainerStarted","Data":"2703f000780ee70b63c1f05502277609b09d73049135e5366c6a0655c8b2aacf"}
Jul 29 20:35:06.471567 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:06.471530    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s" event={"ID":"df5bc3cf-a925-49e8-b0ad-148c62e728d3","Type":"ContainerStarted","Data":"2366dcfd6b924b8e6cad5c40c6d3207727522a87101119f70b6e0d71af36e7e4"}
Jul 29 20:35:06.488659 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:06.488608    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s" podStartSLOduration=0.927803271 podStartE2EDuration="2.488593735s" totalImagesPullingTime="1.560790464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:35:06.486763193 +0000 UTC m=+633.530455368" watchObservedRunningTime="2026-07-29 20:35:06.488593735 +0000 UTC m=+633.532285917"
Jul 29 20:35:11.999228 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:11.999187    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:35:14.625729 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:14.625698    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:14.626099 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:14.625752    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:14.638384 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:14.638356    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:15.524443 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:15.524414    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s"
Jul 29 20:35:17.954311 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:17.954279    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:35:29.156499 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:29.156460    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:35:42.519180 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:42.519143    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:35:45.713475 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:45.713435    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:35:45.813951 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:45.813918    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:35:54.777130 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:35:54.777094    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:36:02.726849 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.726820    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"]
Jul 29 20:36:02.730795 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.730775    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.733321 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.733295    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 29 20:36:02.740929 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.740905    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"]
Jul 29 20:36:02.762028 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.761997    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:36:02.838981 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.838956    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f992bd32-9946-40df-9424-4f9b1cd174ec-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.839089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.838992    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ptr\" (UniqueName: \"kubernetes.io/projected/f992bd32-9946-40df-9424-4f9b1cd174ec-kube-api-access-k5ptr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.839089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.839024    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.839089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.839044    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.839089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.839063    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.839272 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.839169    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.939814 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.939792    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.939928 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.939823    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f992bd32-9946-40df-9424-4f9b1cd174ec-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.939928 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.939851    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ptr\" (UniqueName: \"kubernetes.io/projected/f992bd32-9946-40df-9424-4f9b1cd174ec-kube-api-access-k5ptr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.939928 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.939912    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.940106 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.939957    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.940106 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.939990    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.940218 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.940158    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.940311 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.940288    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.940424 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.940406    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.942125 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.942103    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f992bd32-9946-40df-9424-4f9b1cd174ec-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.942294 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.942276    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f992bd32-9946-40df-9424-4f9b1cd174ec-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:02.946828 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:02.946803    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ptr\" (UniqueName: \"kubernetes.io/projected/f992bd32-9946-40df-9424-4f9b1cd174ec-kube-api-access-k5ptr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4fgx6\" (UID: \"f992bd32-9946-40df-9424-4f9b1cd174ec\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:03.043371 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:03.043301    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:03.166439 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:03.166407    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"]
Jul 29 20:36:03.169342 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:36:03.169298    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf992bd32_9946_40df_9424_4f9b1cd174ec.slice/crio-76338df7f486d59a3f7c48a91fd210ffea8a9eb01ab13da4b246afa8d2404dcc WatchSource:0}: Error finding container 76338df7f486d59a3f7c48a91fd210ffea8a9eb01ab13da4b246afa8d2404dcc: Status 404 returned error can't find the container with id 76338df7f486d59a3f7c48a91fd210ffea8a9eb01ab13da4b246afa8d2404dcc
Jul 29 20:36:03.170796 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:03.170776    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:36:03.746914 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:03.746879    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6" event={"ID":"f992bd32-9946-40df-9424-4f9b1cd174ec","Type":"ContainerStarted","Data":"293da192026bffa7490d1d346ea1b815375738090bf907e137e2fb8d73ddc6e6"}
Jul 29 20:36:03.747259 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:03.746937    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6" event={"ID":"f992bd32-9946-40df-9424-4f9b1cd174ec","Type":"ContainerStarted","Data":"76338df7f486d59a3f7c48a91fd210ffea8a9eb01ab13da4b246afa8d2404dcc"}
Jul 29 20:36:13.043638 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:13.043604    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:13.044089 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:13.043669    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:13.056590 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:13.056564    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:13.073519 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:13.073468    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6" podStartSLOduration=10.886663682 podStartE2EDuration="11.073455299s" totalImagesPullingTime="186.791617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:36:03.763807715 +0000 UTC m=+690.807499898" watchObservedRunningTime="2026-07-29 20:36:13.073455299 +0000 UTC m=+700.117147481"
Jul 29 20:36:13.805487 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:13.805457    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4fgx6"
Jul 29 20:36:19.456535 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.456501    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"]
Jul 29 20:36:19.459967 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.459946    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.462499 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.462476    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 29 20:36:19.471519 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.471497    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"]
Jul 29 20:36:19.475533 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.475512    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pqzx\" (UniqueName: \"kubernetes.io/projected/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-kube-api-access-8pqzx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.475619 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.475556    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.475672 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.475630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.475714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.475676    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.475755 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.475731    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.475755 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.475747    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.501624 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.501601    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:36:19.576490 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.576459    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.576490 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.576489    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.576661 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.576537    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pqzx\" (UniqueName: \"kubernetes.io/projected/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-kube-api-access-8pqzx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.576661 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.576593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.576661 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.576639    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.576816 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.576664    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.577037 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.577011    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.577121 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.577053    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.577121 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.577059    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.578927 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.578902    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.579075 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.579060    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.583841 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.583818    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pqzx\" (UniqueName: \"kubernetes.io/projected/a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea-kube-api-access-8pqzx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm\" (UID: \"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.771926 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.771891    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:19.899693 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:19.899668    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"]
Jul 29 20:36:19.901182 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:36:19.901155    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6d87ff8_aa15_4e2a_aac8_a508ff9eb0ea.slice/crio-eaa47c595e50973a79d1a4f1c1b4cf81894e3b71a446622e94434d28db1a573b WatchSource:0}: Error finding container eaa47c595e50973a79d1a4f1c1b4cf81894e3b71a446622e94434d28db1a573b: Status 404 returned error can't find the container with id eaa47c595e50973a79d1a4f1c1b4cf81894e3b71a446622e94434d28db1a573b
Jul 29 20:36:20.832464 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:20.832424    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm" event={"ID":"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea","Type":"ContainerStarted","Data":"167769a0d6bc1b29876949a9b49203886378861acbfb1d8a95c292d09dacf573"}
Jul 29 20:36:20.832818 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:20.832478    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm" event={"ID":"a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea","Type":"ContainerStarted","Data":"eaa47c595e50973a79d1a4f1c1b4cf81894e3b71a446622e94434d28db1a573b"}
Jul 29 20:36:20.847851 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:20.847800    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm" podStartSLOduration=1.664071228 podStartE2EDuration="1.84778168s" totalImagesPullingTime="183.710452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:36:20.846885732 +0000 UTC m=+707.890577912" watchObservedRunningTime="2026-07-29 20:36:20.84778168 +0000 UTC m=+707.891473863"
Jul 29 20:36:29.772947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:29.772863    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:29.772947 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:29.772940    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:29.785422 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:29.785402    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:29.884512 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:29.884488    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm"
Jul 29 20:36:42.737073 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:42.737034    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:36:42.785619 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:42.785588    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:36:58.340154 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:36:58.340114    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:37:29.401908 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.401874    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84d9f98d8b-8kch5"]
Jul 29 20:37:29.405225 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.405206    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84d9f98d8b-8kch5"
Jul 29 20:37:29.421878 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.421853    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84d9f98d8b-8kch5"]
Jul 29 20:37:29.465368 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.465317    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4280eee-f663-46a7-8ba7-f0f97d53160d-tls-cert\") pod \"authorino-84d9f98d8b-8kch5\" (UID: \"d4280eee-f663-46a7-8ba7-f0f97d53160d\") " pod="kuadrant-system/authorino-84d9f98d8b-8kch5"
Jul 29 20:37:29.465513 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.465491    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjrnz\" (UniqueName: \"kubernetes.io/projected/d4280eee-f663-46a7-8ba7-f0f97d53160d-kube-api-access-jjrnz\") pod \"authorino-84d9f98d8b-8kch5\" (UID: \"d4280eee-f663-46a7-8ba7-f0f97d53160d\") " pod="kuadrant-system/authorino-84d9f98d8b-8kch5"
Jul 29 20:37:29.566903 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.566870    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4280eee-f663-46a7-8ba7-f0f97d53160d-tls-cert\") pod \"authorino-84d9f98d8b-8kch5\" (UID: \"d4280eee-f663-46a7-8ba7-f0f97d53160d\") " pod="kuadrant-system/authorino-84d9f98d8b-8kch5"
Jul 29 20:37:29.567049 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.566978    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjrnz\" (UniqueName: \"kubernetes.io/projected/d4280eee-f663-46a7-8ba7-f0f97d53160d-kube-api-access-jjrnz\") pod \"authorino-84d9f98d8b-8kch5\" (UID: \"d4280eee-f663-46a7-8ba7-f0f97d53160d\") " pod="kuadrant-system/authorino-84d9f98d8b-8kch5"
Jul 29 20:37:29.569402 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.569373    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4280eee-f663-46a7-8ba7-f0f97d53160d-tls-cert\") pod \"authorino-84d9f98d8b-8kch5\" (UID: \"d4280eee-f663-46a7-8ba7-f0f97d53160d\") " pod="kuadrant-system/authorino-84d9f98d8b-8kch5"
Jul 29 20:37:29.578482 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.578458    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjrnz\" (UniqueName: \"kubernetes.io/projected/d4280eee-f663-46a7-8ba7-f0f97d53160d-kube-api-access-jjrnz\") pod \"authorino-84d9f98d8b-8kch5\" (UID: \"d4280eee-f663-46a7-8ba7-f0f97d53160d\") " pod="kuadrant-system/authorino-84d9f98d8b-8kch5"
Jul 29 20:37:29.714671 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.714642    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84d9f98d8b-8kch5"
Jul 29 20:37:29.762648 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.762614    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:37:29.843814 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.843785    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84d9f98d8b-8kch5"]
Jul 29 20:37:29.845812 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:37:29.845776    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4280eee_f663_46a7_8ba7_f0f97d53160d.slice/crio-14d1e33cdb0bca5a374cc0205b1654951319c440fd6ec00a808a31ae7b05c105 WatchSource:0}: Error finding container 14d1e33cdb0bca5a374cc0205b1654951319c440fd6ec00a808a31ae7b05c105: Status 404 returned error can't find the container with id 14d1e33cdb0bca5a374cc0205b1654951319c440fd6ec00a808a31ae7b05c105
Jul 29 20:37:29.846583 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:29.846558    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:37:30.148746 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:30.148666    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84d9f98d8b-8kch5" event={"ID":"d4280eee-f663-46a7-8ba7-f0f97d53160d","Type":"ContainerStarted","Data":"14d1e33cdb0bca5a374cc0205b1654951319c440fd6ec00a808a31ae7b05c105"}
Jul 29 20:37:30.937977 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:30.937885    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:37:31.072036 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.072002    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:37:31.157232 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.157181    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84d9f98d8b-8kch5" event={"ID":"d4280eee-f663-46a7-8ba7-f0f97d53160d","Type":"ContainerStarted","Data":"2c6f725dd3c9b116d09db29a5d2d07d21f0f6334cf9ddb98622cc21b808905eb"}
Jul 29 20:37:31.175782 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.175722    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84d9f98d8b-8kch5" podStartSLOduration=1.487822786 podStartE2EDuration="2.175701275s" totalImagesPullingTime="687.878489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:37:31.172791095 +0000 UTC m=+778.216483279" watchObservedRunningTime="2026-07-29 20:37:31.175701275 +0000 UTC m=+778.219393459"
Jul 29 20:37:31.213699 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.213611    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68f9b67975-hxx9g"]
Jul 29 20:37:31.213871 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.213833    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-68f9b67975-hxx9g" podUID="d3ee8f47-f75b-458e-a119-471f4a0f5120" containerName="authorino" containerID="cri-o://5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483" gracePeriod=30
Jul 29 20:37:31.471740 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.471717    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:37:31.584346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.584308    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3ee8f47-f75b-458e-a119-471f4a0f5120-tls-cert\") pod \"d3ee8f47-f75b-458e-a119-471f4a0f5120\" (UID: \"d3ee8f47-f75b-458e-a119-471f4a0f5120\") "
Jul 29 20:37:31.584528 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.584367    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss5cx\" (UniqueName: \"kubernetes.io/projected/d3ee8f47-f75b-458e-a119-471f4a0f5120-kube-api-access-ss5cx\") pod \"d3ee8f47-f75b-458e-a119-471f4a0f5120\" (UID: \"d3ee8f47-f75b-458e-a119-471f4a0f5120\") "
Jul 29 20:37:31.586463 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.586439    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ee8f47-f75b-458e-a119-471f4a0f5120-kube-api-access-ss5cx" (OuterVolumeSpecName: "kube-api-access-ss5cx") pod "d3ee8f47-f75b-458e-a119-471f4a0f5120" (UID: "d3ee8f47-f75b-458e-a119-471f4a0f5120"). InnerVolumeSpecName "kube-api-access-ss5cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:37:31.594275 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.594249    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ee8f47-f75b-458e-a119-471f4a0f5120-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d3ee8f47-f75b-458e-a119-471f4a0f5120" (UID: "d3ee8f47-f75b-458e-a119-471f4a0f5120"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:37:31.685551 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.685515    2585 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d3ee8f47-f75b-458e-a119-471f4a0f5120-tls-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:37:31.685551 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:31.685545    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss5cx\" (UniqueName: \"kubernetes.io/projected/d3ee8f47-f75b-458e-a119-471f4a0f5120-kube-api-access-ss5cx\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:37:32.164519 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:32.164484    2585 generic.go:356] "Generic (PLEG): container finished" podID="d3ee8f47-f75b-458e-a119-471f4a0f5120" containerID="5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483" exitCode=0
Jul 29 20:37:32.164978 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:32.164532    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f9b67975-hxx9g"
Jul 29 20:37:32.164978 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:32.164530    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f9b67975-hxx9g" event={"ID":"d3ee8f47-f75b-458e-a119-471f4a0f5120","Type":"ContainerDied","Data":"5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483"}
Jul 29 20:37:32.164978 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:32.164642    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f9b67975-hxx9g" event={"ID":"d3ee8f47-f75b-458e-a119-471f4a0f5120","Type":"ContainerDied","Data":"ac7e6be57b1f74202ee6cf6687faccc30a6e4baf2cd226751da7a2323bcd767d"}
Jul 29 20:37:32.164978 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:32.164660    2585 scope.go:117] "RemoveContainer" containerID="5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483"
Jul 29 20:37:32.175748 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:32.175676    2585 scope.go:117] "RemoveContainer" containerID="5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483"
Jul 29 20:37:32.176120 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:37:32.176080    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483\": container with ID starting with 5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483 not found: ID does not exist" containerID="5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483"
Jul 29 20:37:32.176241 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:32.176121    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483"} err="failed to get container status \"5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483\": rpc error: code = NotFound desc = could not find container \"5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483\": container with ID starting with 5c259ec17e94c613f4c95e88f8cdb440aaa9b11d1dfc44372c37054a7248c483 not found: ID does not exist"
Jul 29 20:37:32.191872 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:32.191840    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68f9b67975-hxx9g"]
Jul 29 20:37:32.195908 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:32.195885    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68f9b67975-hxx9g"]
Jul 29 20:37:33.524257 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:33.524225    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ee8f47-f75b-458e-a119-471f4a0f5120" path="/var/lib/kubelet/pods/d3ee8f47-f75b-458e-a119-471f4a0f5120/volumes"
Jul 29 20:37:42.675026 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:37:42.674993    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:38:19.625343 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:19.625298    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5458c6456b-pvxm6"]
Jul 29 20:38:19.625808 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:19.625527    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5458c6456b-pvxm6" podUID="55415b07-f850-4ba7-a5dd-e45780a83e6c" containerName="manager" containerID="cri-o://d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049" gracePeriod=30
Jul 29 20:38:19.870084 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:19.870062    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:38:19.996913 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:19.996886    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55415b07-f850-4ba7-a5dd-e45780a83e6c-webhook-cert\") pod \"55415b07-f850-4ba7-a5dd-e45780a83e6c\" (UID: \"55415b07-f850-4ba7-a5dd-e45780a83e6c\") "
Jul 29 20:38:19.997064 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:19.997000    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs96z\" (UniqueName: \"kubernetes.io/projected/55415b07-f850-4ba7-a5dd-e45780a83e6c-kube-api-access-hs96z\") pod \"55415b07-f850-4ba7-a5dd-e45780a83e6c\" (UID: \"55415b07-f850-4ba7-a5dd-e45780a83e6c\") "
Jul 29 20:38:19.998868 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:19.998829    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55415b07-f850-4ba7-a5dd-e45780a83e6c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "55415b07-f850-4ba7-a5dd-e45780a83e6c" (UID: "55415b07-f850-4ba7-a5dd-e45780a83e6c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:38:19.999003 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:19.998981    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55415b07-f850-4ba7-a5dd-e45780a83e6c-kube-api-access-hs96z" (OuterVolumeSpecName: "kube-api-access-hs96z") pod "55415b07-f850-4ba7-a5dd-e45780a83e6c" (UID: "55415b07-f850-4ba7-a5dd-e45780a83e6c"). InnerVolumeSpecName "kube-api-access-hs96z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:38:20.098094 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.098068    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs96z\" (UniqueName: \"kubernetes.io/projected/55415b07-f850-4ba7-a5dd-e45780a83e6c-kube-api-access-hs96z\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:38:20.098094 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.098092    2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55415b07-f850-4ba7-a5dd-e45780a83e6c-webhook-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:38:20.397044 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.396962    2585 generic.go:356] "Generic (PLEG): container finished" podID="55415b07-f850-4ba7-a5dd-e45780a83e6c" containerID="d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049" exitCode=0
Jul 29 20:38:20.397044 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.397004    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-pvxm6"
Jul 29 20:38:20.397044 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.397023    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-pvxm6" event={"ID":"55415b07-f850-4ba7-a5dd-e45780a83e6c","Type":"ContainerDied","Data":"d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049"}
Jul 29 20:38:20.397263 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.397072    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-pvxm6" event={"ID":"55415b07-f850-4ba7-a5dd-e45780a83e6c","Type":"ContainerDied","Data":"fb9f1096a771af0df1513c62783d1104c880ce810d67756e439b61169d53694b"}
Jul 29 20:38:20.397263 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.397094    2585 scope.go:117] "RemoveContainer" containerID="d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049"
Jul 29 20:38:20.405962 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.405944    2585 scope.go:117] "RemoveContainer" containerID="d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049"
Jul 29 20:38:20.406198 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:38:20.406178    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049\": container with ID starting with d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049 not found: ID does not exist" containerID="d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049"
Jul 29 20:38:20.406243 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.406206    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049"} err="failed to get container status \"d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049\": rpc error: code = NotFound desc = could not find container \"d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049\": container with ID starting with d6b64c837484855e654029c98f487342a0e87d54fb6debb9c8234fd62d8cc049 not found: ID does not exist"
Jul 29 20:38:20.424941 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.424916    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5458c6456b-pvxm6"]
Jul 29 20:38:20.427265 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:20.427245    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5458c6456b-pvxm6"]
Jul 29 20:38:21.524164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:38:21.524133    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55415b07-f850-4ba7-a5dd-e45780a83e6c" path="/var/lib/kubelet/pods/55415b07-f850-4ba7-a5dd-e45780a83e6c/volumes"
Jul 29 20:39:21.220038 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.220002    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5458c6456b-7zkp8"]
Jul 29 20:39:21.220600 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.220577    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ee8f47-f75b-458e-a119-471f4a0f5120" containerName="authorino"
Jul 29 20:39:21.220678 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.220603    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ee8f47-f75b-458e-a119-471f4a0f5120" containerName="authorino"
Jul 29 20:39:21.220678 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.220624    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55415b07-f850-4ba7-a5dd-e45780a83e6c" containerName="manager"
Jul 29 20:39:21.220678 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.220632    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="55415b07-f850-4ba7-a5dd-e45780a83e6c" containerName="manager"
Jul 29 20:39:21.220828 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.220722    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="55415b07-f850-4ba7-a5dd-e45780a83e6c" containerName="manager"
Jul 29 20:39:21.220828 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.220736    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ee8f47-f75b-458e-a119-471f4a0f5120" containerName="authorino"
Jul 29 20:39:21.223853 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.223830    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:21.227446 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.227423    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 20:39:21.227587 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.227424    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ch4nm\""
Jul 29 20:39:21.227587 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.227450    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 20:39:21.230618 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.230592    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5458c6456b-7zkp8"]
Jul 29 20:39:21.302997 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.302968    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-webhook-cert\") pod \"maas-controller-5458c6456b-7zkp8\" (UID: \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\") " pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:21.303097 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.303086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-kube-api-access-n9m5k\") pod \"maas-controller-5458c6456b-7zkp8\" (UID: \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\") " pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:21.403941 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.403902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-kube-api-access-n9m5k\") pod \"maas-controller-5458c6456b-7zkp8\" (UID: \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\") " pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:21.403941 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.403950    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-webhook-cert\") pod \"maas-controller-5458c6456b-7zkp8\" (UID: \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\") " pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:21.406202 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.406186    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-webhook-cert\") pod \"maas-controller-5458c6456b-7zkp8\" (UID: \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\") " pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:21.412697 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.412677    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-kube-api-access-n9m5k\") pod \"maas-controller-5458c6456b-7zkp8\" (UID: \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\") " pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:21.560532 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.560481    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:21.900149 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:21.900119    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5458c6456b-7zkp8"]
Jul 29 20:39:21.901618 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:39:21.901593    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9b94c7_85bd_4c02_8ba0_263308ee7aa0.slice/crio-890e96ac5449435f19db94682e011c33993325f94643cc5a47dc1c701c9e2299 WatchSource:0}: Error finding container 890e96ac5449435f19db94682e011c33993325f94643cc5a47dc1c701c9e2299: Status 404 returned error can't find the container with id 890e96ac5449435f19db94682e011c33993325f94643cc5a47dc1c701c9e2299
Jul 29 20:39:22.690860 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:22.690768    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-7zkp8" event={"ID":"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0","Type":"ContainerStarted","Data":"773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64"}
Jul 29 20:39:22.690860 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:22.690826    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-7zkp8" event={"ID":"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0","Type":"ContainerStarted","Data":"890e96ac5449435f19db94682e011c33993325f94643cc5a47dc1c701c9e2299"}
Jul 29 20:39:22.691363 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:22.690909    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:22.710462 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:22.710412    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5458c6456b-7zkp8" podStartSLOduration=1.267686178 podStartE2EDuration="1.710396966s" totalImagesPullingTime="442.710788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:39:22.708468051 +0000 UTC m=+889.752160236" watchObservedRunningTime="2026-07-29 20:39:22.710396966 +0000 UTC m=+889.754089148"
Jul 29 20:39:33.440938 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:33.440860    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:39:33.442408 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:33.442385    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:39:33.697564 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:33.697490    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:38.255305 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.255274    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:39:38.342069 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.342032    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5458c6456b-7zkp8"]
Jul 29 20:39:38.342238 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.342215    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5458c6456b-7zkp8" podUID="9b9b94c7-85bd-4c02-8ba0-263308ee7aa0" containerName="manager" containerID="cri-o://773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64" gracePeriod=30
Jul 29 20:39:38.584957 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.584933    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:38.660665 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.660634    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-kube-api-access-n9m5k\") pod \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\" (UID: \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\") "
Jul 29 20:39:38.660813 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.660727    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-webhook-cert\") pod \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\" (UID: \"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0\") "
Jul 29 20:39:38.662624 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.662601    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9b9b94c7-85bd-4c02-8ba0-263308ee7aa0" (UID: "9b9b94c7-85bd-4c02-8ba0-263308ee7aa0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:39:38.662747 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.662725    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-kube-api-access-n9m5k" (OuterVolumeSpecName: "kube-api-access-n9m5k") pod "9b9b94c7-85bd-4c02-8ba0-263308ee7aa0" (UID: "9b9b94c7-85bd-4c02-8ba0-263308ee7aa0"). InnerVolumeSpecName "kube-api-access-n9m5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:39:38.764252 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.764172    2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-webhook-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:39:38.764474 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.764457    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0-kube-api-access-n9m5k\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:39:38.767343 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.767299    2585 generic.go:356] "Generic (PLEG): container finished" podID="9b9b94c7-85bd-4c02-8ba0-263308ee7aa0" containerID="773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64" exitCode=0
Jul 29 20:39:38.767468 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.767433    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-7zkp8" event={"ID":"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0","Type":"ContainerDied","Data":"773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64"}
Jul 29 20:39:38.767527 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.767469    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-7zkp8"
Jul 29 20:39:38.767750 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.767725    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-7zkp8" event={"ID":"9b9b94c7-85bd-4c02-8ba0-263308ee7aa0","Type":"ContainerDied","Data":"890e96ac5449435f19db94682e011c33993325f94643cc5a47dc1c701c9e2299"}
Jul 29 20:39:38.767894 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.767882    2585 scope.go:117] "RemoveContainer" containerID="773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64"
Jul 29 20:39:38.776601 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.776584    2585 scope.go:117] "RemoveContainer" containerID="773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64"
Jul 29 20:39:38.776834 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:39:38.776818    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64\": container with ID starting with 773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64 not found: ID does not exist" containerID="773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64"
Jul 29 20:39:38.776883 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.776840    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64"} err="failed to get container status \"773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64\": rpc error: code = NotFound desc = could not find container \"773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64\": container with ID starting with 773a5f82c2efd2cff3bafd2a681af63d5ceadd5bfeece5862b22bb6b6ba9dd64 not found: ID does not exist"
Jul 29 20:39:38.788525 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.788502    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5458c6456b-7zkp8"]
Jul 29 20:39:38.791412 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:38.791392    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5458c6456b-7zkp8"]
Jul 29 20:39:39.436943 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.436904    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:39:39.524325 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.524292    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b9b94c7-85bd-4c02-8ba0-263308ee7aa0" path="/var/lib/kubelet/pods/9b9b94c7-85bd-4c02-8ba0-263308ee7aa0/volumes"
Jul 29 20:39:39.552648 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.552617    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5458c6456b-hbrtq"]
Jul 29 20:39:39.552978 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.552966    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b9b94c7-85bd-4c02-8ba0-263308ee7aa0" containerName="manager"
Jul 29 20:39:39.553018 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.552980    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b9b94c7-85bd-4c02-8ba0-263308ee7aa0" containerName="manager"
Jul 29 20:39:39.553074 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.553064    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b9b94c7-85bd-4c02-8ba0-263308ee7aa0" containerName="manager"
Jul 29 20:39:39.557364 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.557322    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:39:39.561180 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.561161    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 20:39:39.561293 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.561161    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ch4nm\""
Jul 29 20:39:39.561293 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.561163    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 20:39:39.566119 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.566094    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5458c6456b-hbrtq"]
Jul 29 20:39:39.672355 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.672300    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdsr\" (UniqueName: \"kubernetes.io/projected/9659f107-7c42-47d6-b32e-8b5ca15a2335-kube-api-access-vbdsr\") pod \"maas-controller-5458c6456b-hbrtq\" (UID: \"9659f107-7c42-47d6-b32e-8b5ca15a2335\") " pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:39:39.672492 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.672408    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9659f107-7c42-47d6-b32e-8b5ca15a2335-webhook-cert\") pod \"maas-controller-5458c6456b-hbrtq\" (UID: \"9659f107-7c42-47d6-b32e-8b5ca15a2335\") " pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:39:39.772880 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.772850    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdsr\" (UniqueName: \"kubernetes.io/projected/9659f107-7c42-47d6-b32e-8b5ca15a2335-kube-api-access-vbdsr\") pod \"maas-controller-5458c6456b-hbrtq\" (UID: \"9659f107-7c42-47d6-b32e-8b5ca15a2335\") " pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:39:39.773066 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.772917    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9659f107-7c42-47d6-b32e-8b5ca15a2335-webhook-cert\") pod \"maas-controller-5458c6456b-hbrtq\" (UID: \"9659f107-7c42-47d6-b32e-8b5ca15a2335\") " pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:39:39.775388 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.775365    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9659f107-7c42-47d6-b32e-8b5ca15a2335-webhook-cert\") pod \"maas-controller-5458c6456b-hbrtq\" (UID: \"9659f107-7c42-47d6-b32e-8b5ca15a2335\") " pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:39:39.780999 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.780975    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdsr\" (UniqueName: \"kubernetes.io/projected/9659f107-7c42-47d6-b32e-8b5ca15a2335-kube-api-access-vbdsr\") pod \"maas-controller-5458c6456b-hbrtq\" (UID: \"9659f107-7c42-47d6-b32e-8b5ca15a2335\") " pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:39:39.868206 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.868177    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:39:39.996618 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:39.996594    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5458c6456b-hbrtq"]
Jul 29 20:39:39.997977 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:39:39.997954    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9659f107_7c42_47d6_b32e_8b5ca15a2335.slice/crio-80edfc60a466e4a6bc325553f32975196d6cb88eafd441a555611e0256c82d44 WatchSource:0}: Error finding container 80edfc60a466e4a6bc325553f32975196d6cb88eafd441a555611e0256c82d44: Status 404 returned error can't find the container with id 80edfc60a466e4a6bc325553f32975196d6cb88eafd441a555611e0256c82d44
Jul 29 20:39:40.783094 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:40.783056    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-hbrtq" event={"ID":"9659f107-7c42-47d6-b32e-8b5ca15a2335","Type":"ContainerStarted","Data":"0fff60b0814303d20e2d352b9b284bf7310ba9b3a1b0c607f2a378f49e090dff"}
Jul 29 20:39:40.783531 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:40.783120    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5458c6456b-hbrtq" event={"ID":"9659f107-7c42-47d6-b32e-8b5ca15a2335","Type":"ContainerStarted","Data":"80edfc60a466e4a6bc325553f32975196d6cb88eafd441a555611e0256c82d44"}
Jul 29 20:39:40.783531 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:40.783229    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:39:40.803429 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:40.803382    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5458c6456b-hbrtq" podStartSLOduration=1.4380857329999999 podStartE2EDuration="1.803367921s" totalImagesPullingTime="365.282188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:39:40.80164229 +0000 UTC m=+907.845334474" watchObservedRunningTime="2026-07-29 20:39:40.803367921 +0000 UTC m=+907.847060104"
Jul 29 20:39:51.790309 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:39:51.790275    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5458c6456b-hbrtq"
Jul 29 20:40:32.865081 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:32.865046    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:32.963084 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:32.963050    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:33.063475 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:33.063445    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:38.914889 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:38.914855    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:39.015563 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:39.015532    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:39.117694 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:39.117662    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:39.315506 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:39.315475    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:44.885844 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:44.885807    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:45.906050 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:45.906019    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:46.011926 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:46.011895    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:55.932295 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:55.932262    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:40:55.982694 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:40:55.982667    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:44:01.140800 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:44:01.140727    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:44:33.477014 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:44:33.476985    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:44:33.478390 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:44:33.478366    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:44:43.946476 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:44:43.946441    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:44:43.994709 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:44:43.994677    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:46:06.932569 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:46:06.932526    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:48:03.027511 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:48:03.027466    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:48:03.117046 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:48:03.117009    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:49:33.508775 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:49:33.508749    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:49:33.512971 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:49:33.512950    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:50:19.143198 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:19.143106    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:50:21.121209 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.121168    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"]
Jul 29 20:50:21.125501 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.125473    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.128144 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.128120    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d456fab9-kserve-self-signed-certs\""
Jul 29 20:50:21.143377 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.143306    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"]
Jul 29 20:50:21.147536 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.147507    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:50:21.295895 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.295872    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-home\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.296008 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.295901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tmp-dir\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.296008 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.295936    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tls-certs\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.296098 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.296011    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-dshm\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.296098 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.296043    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmjj\" (UniqueName: \"kubernetes.io/projected/3a8fc6e0-c91b-4f90-af58-9fefa2096702-kube-api-access-xnmjj\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.296098 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.296067    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-model-cache\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.397290 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.397228    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-dshm\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.397290 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.397263    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmjj\" (UniqueName: \"kubernetes.io/projected/3a8fc6e0-c91b-4f90-af58-9fefa2096702-kube-api-access-xnmjj\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.397290 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.397286    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-model-cache\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.397482 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.397360    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-home\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.397482 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.397382    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tmp-dir\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.397482 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.397432    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tls-certs\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.397797 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.397773    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-model-cache\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.397871 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.397789    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-home\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.397871 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.397834    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tmp-dir\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.399488 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.399467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-dshm\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.399807 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.399782    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tls-certs\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.405471 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.405450    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmjj\" (UniqueName: \"kubernetes.io/projected/3a8fc6e0-c91b-4f90-af58-9fefa2096702-kube-api-access-xnmjj\") pod \"e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") " pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.436875 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.436857    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:21.565426 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.565396    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"]
Jul 29 20:50:21.567279 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:50:21.567247    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a8fc6e0_c91b_4f90_af58_9fefa2096702.slice/crio-79354fc1811e361b9e77d46e6298a9e1883e50c50027d8bd2de0b50cbc9ffdde WatchSource:0}: Error finding container 79354fc1811e361b9e77d46e6298a9e1883e50c50027d8bd2de0b50cbc9ffdde: Status 404 returned error can't find the container with id 79354fc1811e361b9e77d46e6298a9e1883e50c50027d8bd2de0b50cbc9ffdde
Jul 29 20:50:21.836629 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.836585    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6" event={"ID":"3a8fc6e0-c91b-4f90-af58-9fefa2096702","Type":"ContainerStarted","Data":"7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438"}
Jul 29 20:50:21.836793 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.836639    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6" event={"ID":"3a8fc6e0-c91b-4f90-af58-9fefa2096702","Type":"ContainerStarted","Data":"79354fc1811e361b9e77d46e6298a9e1883e50c50027d8bd2de0b50cbc9ffdde"}
Jul 29 20:50:21.854163 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:21.854114    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6" podStartSLOduration=0.854101378 podStartE2EDuration="854.101378ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:50:21.85309387 +0000 UTC m=+1548.896786054" watchObservedRunningTime="2026-07-29 20:50:21.854101378 +0000 UTC m=+1548.897793560"
Jul 29 20:50:31.438014 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:31.437928    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:31.438014 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:31.437985    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:31.450621 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:31.450593    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:31.901085 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:31.901051    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:50:43.635139 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:50:43.635104    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:51:10.756553 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:10.756513    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"]
Jul 29 20:51:10.779287 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:10.779255    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:51:12.052322 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:12.052285    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:51:25.775983 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:25.775916    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6" podUID="3a8fc6e0-c91b-4f90-af58-9fefa2096702" containerName="main" containerID="cri-o://7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438" gracePeriod=45
Jul 29 20:51:26.014688 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.014657    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:51:26.122082 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122008    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-home\") pod \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") "
Jul 29 20:51:26.122247 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122087    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tmp-dir\") pod \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") "
Jul 29 20:51:26.122247 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122120    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnmjj\" (UniqueName: \"kubernetes.io/projected/3a8fc6e0-c91b-4f90-af58-9fefa2096702-kube-api-access-xnmjj\") pod \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") "
Jul 29 20:51:26.122247 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122165    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tls-certs\") pod \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") "
Jul 29 20:51:26.122247 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122206    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-dshm\") pod \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") "
Jul 29 20:51:26.122247 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122232    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-model-cache\") pod \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\" (UID: \"3a8fc6e0-c91b-4f90-af58-9fefa2096702\") "
Jul 29 20:51:26.122570 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122289    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-home" (OuterVolumeSpecName: "home") pod "3a8fc6e0-c91b-4f90-af58-9fefa2096702" (UID: "3a8fc6e0-c91b-4f90-af58-9fefa2096702"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:51:26.122570 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122373    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a8fc6e0-c91b-4f90-af58-9fefa2096702" (UID: "3a8fc6e0-c91b-4f90-af58-9fefa2096702"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:51:26.122570 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122547    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:51:26.122570 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122549    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a8fc6e0-c91b-4f90-af58-9fefa2096702" (UID: "3a8fc6e0-c91b-4f90-af58-9fefa2096702"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:51:26.122570 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.122559    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:51:26.124474 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.124448    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a8fc6e0-c91b-4f90-af58-9fefa2096702" (UID: "3a8fc6e0-c91b-4f90-af58-9fefa2096702"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:51:26.124576 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.124462    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-dshm" (OuterVolumeSpecName: "dshm") pod "3a8fc6e0-c91b-4f90-af58-9fefa2096702" (UID: "3a8fc6e0-c91b-4f90-af58-9fefa2096702"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:51:26.124576 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.124525    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8fc6e0-c91b-4f90-af58-9fefa2096702-kube-api-access-xnmjj" (OuterVolumeSpecName: "kube-api-access-xnmjj") pod "3a8fc6e0-c91b-4f90-af58-9fefa2096702" (UID: "3a8fc6e0-c91b-4f90-af58-9fefa2096702"). InnerVolumeSpecName "kube-api-access-xnmjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:51:26.154699 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.154670    2585 generic.go:356] "Generic (PLEG): container finished" podID="3a8fc6e0-c91b-4f90-af58-9fefa2096702" containerID="7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438" exitCode=0
Jul 29 20:51:26.154788 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.154712    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6" event={"ID":"3a8fc6e0-c91b-4f90-af58-9fefa2096702","Type":"ContainerDied","Data":"7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438"}
Jul 29 20:51:26.154788 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.154745    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6" event={"ID":"3a8fc6e0-c91b-4f90-af58-9fefa2096702","Type":"ContainerDied","Data":"79354fc1811e361b9e77d46e6298a9e1883e50c50027d8bd2de0b50cbc9ffdde"}
Jul 29 20:51:26.154788 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.154760    2585 scope.go:117] "RemoveContainer" containerID="7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438"
Jul 29 20:51:26.154911 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.154718    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"
Jul 29 20:51:26.163911 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.163894    2585 scope.go:117] "RemoveContainer" containerID="7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438"
Jul 29 20:51:26.164148 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:51:26.164127    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438\": container with ID starting with 7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438 not found: ID does not exist" containerID="7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438"
Jul 29 20:51:26.164241 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.164156    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438"} err="failed to get container status \"7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438\": rpc error: code = NotFound desc = could not find container \"7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438\": container with ID starting with 7cb7da348214c6e0f1e751524fd64701f42f857171587e03dd89a6950a1ac438 not found: ID does not exist"
Jul 29 20:51:26.176856 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.176835    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"]
Jul 29 20:51:26.180681 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.180657    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d456fab9-kserve-5ccc65c64c-4t9r6"]
Jul 29 20:51:26.223878 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.223855    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmjj\" (UniqueName: \"kubernetes.io/projected/3a8fc6e0-c91b-4f90-af58-9fefa2096702-kube-api-access-xnmjj\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:51:26.223878 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.223877    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a8fc6e0-c91b-4f90-af58-9fefa2096702-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:51:26.224004 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.223886    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:51:26.224004 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:26.223894    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a8fc6e0-c91b-4f90-af58-9fefa2096702-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:51:27.523844 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:51:27.523811    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a8fc6e0-c91b-4f90-af58-9fefa2096702" path="/var/lib/kubelet/pods/3a8fc6e0-c91b-4f90-af58-9fefa2096702/volumes"
Jul 29 20:52:28.944286 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:52:28.944243    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:53:49.929073 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:53:49.929037    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:54:33.539098 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:54:33.539070    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:54:33.544392 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:54:33.544366    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:55:02.069502 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:02.069465    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:38.541449 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:38.541413    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:41.650748 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:41.650714    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:41.751157 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:41.751133    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:42.064557 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:42.064521    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:47.702033 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:47.701995    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:47.759820 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:47.759777    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:47.935446 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:47.935408    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:48.237448 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:48.237419    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:48.643385 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:48.643277    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:50.305952 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.305911    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:50.400212 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.400182    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:50.522388 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.522355    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"]
Jul 29 20:55:50.522919 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.522903    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8fc6e0-c91b-4f90-af58-9fefa2096702" containerName="main"
Jul 29 20:55:50.522999 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.522921    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8fc6e0-c91b-4f90-af58-9fefa2096702" containerName="main"
Jul 29 20:55:50.523038 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.523019    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a8fc6e0-c91b-4f90-af58-9fefa2096702" containerName="main"
Jul 29 20:55:50.526260 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.526244    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.529114 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.529093    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a64c614d-serving-cert\""
Jul 29 20:55:50.537276 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.537246    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"]
Jul 29 20:55:50.560581 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.560515    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:50.666171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.666150    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e74e4abb-70be-4c37-a817-055b38503183-openshift-service-ca\") pod \"maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.666282 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.666178    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fldlp\" (UniqueName: \"kubernetes.io/projected/e74e4abb-70be-4c37-a817-055b38503183-kube-api-access-fldlp\") pod \"maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.666282 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.666203    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e74e4abb-70be-4c37-a817-055b38503183-maas-api-tls\") pod \"maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.767022 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.766996    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e74e4abb-70be-4c37-a817-055b38503183-openshift-service-ca\") pod \"maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.767121 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.767025    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fldlp\" (UniqueName: \"kubernetes.io/projected/e74e4abb-70be-4c37-a817-055b38503183-kube-api-access-fldlp\") pod \"maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.767121 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.767054    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e74e4abb-70be-4c37-a817-055b38503183-maas-api-tls\") pod \"maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.767745 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.767724    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e74e4abb-70be-4c37-a817-055b38503183-openshift-service-ca\") pod \"maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.769017 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.768996    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:50.769576 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.769544    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e74e4abb-70be-4c37-a817-055b38503183-maas-api-tls\") pod \"maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.775043 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.775020    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fldlp\" (UniqueName: \"kubernetes.io/projected/e74e4abb-70be-4c37-a817-055b38503183-kube-api-access-fldlp\") pod \"maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.838747 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.838698    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:50.966465 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.966414    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"]
Jul 29 20:55:50.967703 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:55:50.967679    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74e4abb_70be_4c37_a817_055b38503183.slice/crio-a78695f2f93301026e29973ded1616bd44dfe251c810d6b342fc913428b9dd13 WatchSource:0}: Error finding container a78695f2f93301026e29973ded1616bd44dfe251c810d6b342fc913428b9dd13: Status 404 returned error can't find the container with id a78695f2f93301026e29973ded1616bd44dfe251c810d6b342fc913428b9dd13
Jul 29 20:55:50.969130 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:50.969111    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:55:51.127805 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:51.127734    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:51.229055 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:51.229021    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:55:51.439150 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:51.439124    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp" event={"ID":"e74e4abb-70be-4c37-a817-055b38503183","Type":"ContainerStarted","Data":"a78695f2f93301026e29973ded1616bd44dfe251c810d6b342fc913428b9dd13"}
Jul 29 20:55:52.446484 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:52.446441    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp" event={"ID":"e74e4abb-70be-4c37-a817-055b38503183","Type":"ContainerStarted","Data":"b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a"}
Jul 29 20:55:52.446847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:52.446521    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:55:52.464186 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:52.464138    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp" podStartSLOduration=2.063486801 podStartE2EDuration="2.464122551s" totalImagesPullingTime="400.63575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:52.462239001 +0000 UTC m=+1879.505931186" watchObservedRunningTime="2026-07-29 20:55:52.464122551 +0000 UTC m=+1879.507814734"
Jul 29 20:55:58.457045 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:55:58.456968    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:56:31.025385 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.025351    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"]
Jul 29 20:56:31.025914 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.025582    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp" podUID="e74e4abb-70be-4c37-a817-055b38503183" containerName="maas-api" containerID="cri-o://b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a" gracePeriod=30
Jul 29 20:56:31.054017 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.053985    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:56:31.285784 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.285724    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:56:31.397158 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.397126    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fldlp\" (UniqueName: \"kubernetes.io/projected/e74e4abb-70be-4c37-a817-055b38503183-kube-api-access-fldlp\") pod \"e74e4abb-70be-4c37-a817-055b38503183\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") "
Jul 29 20:56:31.397320 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.397173    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e74e4abb-70be-4c37-a817-055b38503183-openshift-service-ca\") pod \"e74e4abb-70be-4c37-a817-055b38503183\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") "
Jul 29 20:56:31.397320 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.397285    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e74e4abb-70be-4c37-a817-055b38503183-maas-api-tls\") pod \"e74e4abb-70be-4c37-a817-055b38503183\" (UID: \"e74e4abb-70be-4c37-a817-055b38503183\") "
Jul 29 20:56:31.397615 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.397590    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74e4abb-70be-4c37-a817-055b38503183-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e74e4abb-70be-4c37-a817-055b38503183" (UID: "e74e4abb-70be-4c37-a817-055b38503183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:56:31.399393 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.399358    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74e4abb-70be-4c37-a817-055b38503183-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e74e4abb-70be-4c37-a817-055b38503183" (UID: "e74e4abb-70be-4c37-a817-055b38503183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:56:31.399500 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.399447    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74e4abb-70be-4c37-a817-055b38503183-kube-api-access-fldlp" (OuterVolumeSpecName: "kube-api-access-fldlp") pod "e74e4abb-70be-4c37-a817-055b38503183" (UID: "e74e4abb-70be-4c37-a817-055b38503183"). InnerVolumeSpecName "kube-api-access-fldlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:56:31.498925 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.498900    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e74e4abb-70be-4c37-a817-055b38503183-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:56:31.498925 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.498923    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fldlp\" (UniqueName: \"kubernetes.io/projected/e74e4abb-70be-4c37-a817-055b38503183-kube-api-access-fldlp\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:56:31.499061 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.498934    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e74e4abb-70be-4c37-a817-055b38503183-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:56:31.633539 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.633476    2585 generic.go:356] "Generic (PLEG): container finished" podID="e74e4abb-70be-4c37-a817-055b38503183" containerID="b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a" exitCode=0
Jul 29 20:56:31.633539 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.633527    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"
Jul 29 20:56:31.633539 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.633536    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp" event={"ID":"e74e4abb-70be-4c37-a817-055b38503183","Type":"ContainerDied","Data":"b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a"}
Jul 29 20:56:31.633771 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.633584    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp" event={"ID":"e74e4abb-70be-4c37-a817-055b38503183","Type":"ContainerDied","Data":"a78695f2f93301026e29973ded1616bd44dfe251c810d6b342fc913428b9dd13"}
Jul 29 20:56:31.633771 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.633605    2585 scope.go:117] "RemoveContainer" containerID="b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a"
Jul 29 20:56:31.642101 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.642083    2585 scope.go:117] "RemoveContainer" containerID="b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a"
Jul 29 20:56:31.642416 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:56:31.642394    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a\": container with ID starting with b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a not found: ID does not exist" containerID="b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a"
Jul 29 20:56:31.642490 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.642422    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a"} err="failed to get container status \"b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a\": rpc error: code = NotFound desc = could not find container \"b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a\": container with ID starting with b491efc5bc24f253706acfc336627820fc420239bdc25d9ef167e4a0fe15a79a not found: ID does not exist"
Jul 29 20:56:31.650461 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.650440    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"]
Jul 29 20:56:31.655811 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:31.655786    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a64c614d-5fcdc77c6b-ctvlp"]
Jul 29 20:56:32.666024 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:32.665992    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:56:32.975041 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:32.975006    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:56:33.524701 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:33.524672    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e74e4abb-70be-4c37-a817-055b38503183" path="/var/lib/kubelet/pods/e74e4abb-70be-4c37-a817-055b38503183/volumes"
Jul 29 20:56:34.783559 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.783529    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:56:34.847560 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.847530    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"]
Jul 29 20:56:34.848039 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.848022    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74e4abb-70be-4c37-a817-055b38503183" containerName="maas-api"
Jul 29 20:56:34.848039 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.848040    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74e4abb-70be-4c37-a817-055b38503183" containerName="maas-api"
Jul 29 20:56:34.848202 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.848127    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74e4abb-70be-4c37-a817-055b38503183" containerName="maas-api"
Jul 29 20:56:34.852661 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.852641    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:34.855381 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.855361    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-eba524b8-serving-cert\""
Jul 29 20:56:34.859362 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.859325    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"]
Jul 29 20:56:34.874911 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.874891    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:56:34.924019 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.923991    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-openshift-service-ca\") pod \"maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:34.924105 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.924030    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-maas-api-tls\") pod \"maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:34.924148 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:34.924093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twl9b\" (UniqueName: \"kubernetes.io/projected/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-kube-api-access-twl9b\") pod \"maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:35.011519 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.011492    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:56:35.024645 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.024622    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twl9b\" (UniqueName: \"kubernetes.io/projected/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-kube-api-access-twl9b\") pod \"maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:35.024759 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.024719    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-openshift-service-ca\") pod \"maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:35.024823 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.024772    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-maas-api-tls\") pod \"maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:35.025320 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.025300    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-openshift-service-ca\") pod \"maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:35.027161 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.027131    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-maas-api-tls\") pod \"maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:35.032352 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.032320    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twl9b\" (UniqueName: \"kubernetes.io/projected/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-kube-api-access-twl9b\") pod \"maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:35.164833 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.164774    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:35.476585 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.476556    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:56:35.509937 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.503064    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"]
Jul 29 20:56:35.511608 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:56:35.511581    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358e9a38_c1b6_4f14_9eb1_85d4703a55f9.slice/crio-46445ed77ce266a1b82466ea40152e06e07fde6e59ebbf1ea60a58c181d72c62 WatchSource:0}: Error finding container 46445ed77ce266a1b82466ea40152e06e07fde6e59ebbf1ea60a58c181d72c62: Status 404 returned error can't find the container with id 46445ed77ce266a1b82466ea40152e06e07fde6e59ebbf1ea60a58c181d72c62
Jul 29 20:56:35.574741 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.574717    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:56:35.656634 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:35.656597    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts" event={"ID":"358e9a38-c1b6-4f14-9eb1-85d4703a55f9","Type":"ContainerStarted","Data":"46445ed77ce266a1b82466ea40152e06e07fde6e59ebbf1ea60a58c181d72c62"}
Jul 29 20:56:36.662971 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:36.662935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts" event={"ID":"358e9a38-c1b6-4f14-9eb1-85d4703a55f9","Type":"ContainerStarted","Data":"dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5"}
Jul 29 20:56:36.663482 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:36.663070    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:56:36.680999 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:36.680946    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts" podStartSLOduration=2.355810968 podStartE2EDuration="2.680926175s" totalImagesPullingTime="325.115207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:36.680125899 +0000 UTC m=+1923.723818083" watchObservedRunningTime="2026-07-29 20:56:36.680926175 +0000 UTC m=+1923.724618359"
Jul 29 20:56:42.674598 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:56:42.674563    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:57:06.212390 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.212359    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"]
Jul 29 20:57:06.212882 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.212566    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts" podUID="358e9a38-c1b6-4f14-9eb1-85d4703a55f9" containerName="maas-api" containerID="cri-o://dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5" gracePeriod=30
Jul 29 20:57:06.462944 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.462888    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:57:06.478960 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.478939    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-openshift-service-ca\") pod \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") "
Jul 29 20:57:06.479074 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.478979    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twl9b\" (UniqueName: \"kubernetes.io/projected/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-kube-api-access-twl9b\") pod \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") "
Jul 29 20:57:06.479074 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.479023    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-maas-api-tls\") pod \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\" (UID: \"358e9a38-c1b6-4f14-9eb1-85d4703a55f9\") "
Jul 29 20:57:06.479319 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.479297    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "358e9a38-c1b6-4f14-9eb1-85d4703a55f9" (UID: "358e9a38-c1b6-4f14-9eb1-85d4703a55f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:57:06.479472 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.479455    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:57:06.481519 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.481485    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "358e9a38-c1b6-4f14-9eb1-85d4703a55f9" (UID: "358e9a38-c1b6-4f14-9eb1-85d4703a55f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:57:06.481639 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.481579    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-kube-api-access-twl9b" (OuterVolumeSpecName: "kube-api-access-twl9b") pod "358e9a38-c1b6-4f14-9eb1-85d4703a55f9" (UID: "358e9a38-c1b6-4f14-9eb1-85d4703a55f9"). InnerVolumeSpecName "kube-api-access-twl9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:57:06.534672 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.534638    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:06.579962 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.579935    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twl9b\" (UniqueName: \"kubernetes.io/projected/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-kube-api-access-twl9b\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:57:06.579962 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.579960    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358e9a38-c1b6-4f14-9eb1-85d4703a55f9-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:57:06.614714 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.614691    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:06.670892 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.670870    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:06.813081 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.813046    2585 generic.go:356] "Generic (PLEG): container finished" podID="358e9a38-c1b6-4f14-9eb1-85d4703a55f9" containerID="dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5" exitCode=0
Jul 29 20:57:06.813211 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.813090    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"
Jul 29 20:57:06.813211 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.813121    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts" event={"ID":"358e9a38-c1b6-4f14-9eb1-85d4703a55f9","Type":"ContainerDied","Data":"dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5"}
Jul 29 20:57:06.813211 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.813160    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts" event={"ID":"358e9a38-c1b6-4f14-9eb1-85d4703a55f9","Type":"ContainerDied","Data":"46445ed77ce266a1b82466ea40152e06e07fde6e59ebbf1ea60a58c181d72c62"}
Jul 29 20:57:06.813211 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.813180    2585 scope.go:117] "RemoveContainer" containerID="dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5"
Jul 29 20:57:06.822527 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.822510    2585 scope.go:117] "RemoveContainer" containerID="dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5"
Jul 29 20:57:06.822791 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:57:06.822774    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5\": container with ID starting with dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5 not found: ID does not exist" containerID="dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5"
Jul 29 20:57:06.822847 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.822802    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5"} err="failed to get container status \"dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5\": rpc error: code = NotFound desc = could not find container \"dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5\": container with ID starting with dc6b286c6265357de27977ca71dbd7f22166d87fb0b409cfa328d173aebe7be5 not found: ID does not exist"
Jul 29 20:57:06.865926 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.865901    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"]
Jul 29 20:57:06.869558 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:06.869537    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eba524b8-84fcf74577-4pdts"]
Jul 29 20:57:07.004314 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:07.004285    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:07.216305 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:07.216277    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:07.526048 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:07.525975    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358e9a38-c1b6-4f14-9eb1-85d4703a55f9" path="/var/lib/kubelet/pods/358e9a38-c1b6-4f14-9eb1-85d4703a55f9/volumes"
Jul 29 20:57:07.588844 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:07.588811    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:07.967777 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:07.967743    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:09.845723 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:09.845695    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:09.911305 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:09.911276    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:10.011642 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.011613    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"]
Jul 29 20:57:10.012385 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.012358    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358e9a38-c1b6-4f14-9eb1-85d4703a55f9" containerName="maas-api"
Jul 29 20:57:10.012385 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.012383    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="358e9a38-c1b6-4f14-9eb1-85d4703a55f9" containerName="maas-api"
Jul 29 20:57:10.012572 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.012492    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="358e9a38-c1b6-4f14-9eb1-85d4703a55f9" containerName="maas-api"
Jul 29 20:57:10.016106 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.016084    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.018924 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.018899    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7cdd0d7d-serving-cert\""
Jul 29 20:57:10.023982 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.023958    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"]
Jul 29 20:57:10.109191 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.109130    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7003c4d0-2364-4ef1-9168-953b601b8278-openshift-service-ca\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.109191 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.109162    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjg4\" (UniqueName: \"kubernetes.io/projected/7003c4d0-2364-4ef1-9168-953b601b8278-kube-api-access-2qjg4\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.109383 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.109316    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7003c4d0-2364-4ef1-9168-953b601b8278-maas-api-tls\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.209886 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.209851    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7003c4d0-2364-4ef1-9168-953b601b8278-maas-api-tls\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.210010 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.209921    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7003c4d0-2364-4ef1-9168-953b601b8278-openshift-service-ca\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.210010 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.209954    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjg4\" (UniqueName: \"kubernetes.io/projected/7003c4d0-2364-4ef1-9168-953b601b8278-kube-api-access-2qjg4\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.210127 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:57:10.210011    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-serving-cert: secret "maas-api-e2e-ait-7cdd0d7d-serving-cert" not found
Jul 29 20:57:10.210127 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:57:10.210077    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7003c4d0-2364-4ef1-9168-953b601b8278-maas-api-tls podName:7003c4d0-2364-4ef1-9168-953b601b8278 nodeName:}" failed. No retries permitted until 2026-07-29 20:57:10.710055066 +0000 UTC m=+1957.753747241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7003c4d0-2364-4ef1-9168-953b601b8278-maas-api-tls") pod "maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx" (UID: "7003c4d0-2364-4ef1-9168-953b601b8278") : secret "maas-api-e2e-ait-7cdd0d7d-serving-cert" not found
Jul 29 20:57:10.210575 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.210557    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7003c4d0-2364-4ef1-9168-953b601b8278-openshift-service-ca\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.222224 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.222198    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjg4\" (UniqueName: \"kubernetes.io/projected/7003c4d0-2364-4ef1-9168-953b601b8278-kube-api-access-2qjg4\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.433010 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.432938    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:10.618817 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.618784    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:10.712455 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.712382    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7003c4d0-2364-4ef1-9168-953b601b8278-maas-api-tls\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.714783 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.714757    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7003c4d0-2364-4ef1-9168-953b601b8278-maas-api-tls\") pod \"maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:10.715569 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.715550    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:10.931356 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:10.931304    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:11.058526 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:11.058499    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"]
Jul 29 20:57:11.060218 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:57:11.060193    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7003c4d0_2364_4ef1_9168_953b601b8278.slice/crio-5978600cbf1fc4aea644ff350c6d5be351345750e8138a039dbd6877317d94a6 WatchSource:0}: Error finding container 5978600cbf1fc4aea644ff350c6d5be351345750e8138a039dbd6877317d94a6: Status 404 returned error can't find the container with id 5978600cbf1fc4aea644ff350c6d5be351345750e8138a039dbd6877317d94a6
Jul 29 20:57:11.844295 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:11.844255    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx" event={"ID":"7003c4d0-2364-4ef1-9168-953b601b8278","Type":"ContainerStarted","Data":"12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0"}
Jul 29 20:57:11.844476 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:11.844307    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx" event={"ID":"7003c4d0-2364-4ef1-9168-953b601b8278","Type":"ContainerStarted","Data":"5978600cbf1fc4aea644ff350c6d5be351345750e8138a039dbd6877317d94a6"}
Jul 29 20:57:11.844476 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:11.844383    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:11.862802 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:11.862759    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx" podStartSLOduration=2.483876326 podStartE2EDuration="2.862744669s" totalImagesPullingTime="378.868343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:57:11.859870252 +0000 UTC m=+1958.903562447" watchObservedRunningTime="2026-07-29 20:57:11.862744669 +0000 UTC m=+1958.906436851"
Jul 29 20:57:17.853444 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:17.853417    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:40.536317 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.536278    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"]
Jul 29 20:57:40.536862 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.536569    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx" podUID="7003c4d0-2364-4ef1-9168-953b601b8278" containerName="maas-api" containerID="cri-o://12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0" gracePeriod=30
Jul 29 20:57:40.568311 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.568271    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:40.668065 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.668035    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:40.738448 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.738420    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:40.787195 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.787139    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:40.795283 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.795256    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:40.870074 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.870046    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7003c4d0-2364-4ef1-9168-953b601b8278-openshift-service-ca\") pod \"7003c4d0-2364-4ef1-9168-953b601b8278\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") "
Jul 29 20:57:40.870216 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.870198    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7003c4d0-2364-4ef1-9168-953b601b8278-maas-api-tls\") pod \"7003c4d0-2364-4ef1-9168-953b601b8278\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") "
Jul 29 20:57:40.870271 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.870236    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qjg4\" (UniqueName: \"kubernetes.io/projected/7003c4d0-2364-4ef1-9168-953b601b8278-kube-api-access-2qjg4\") pod \"7003c4d0-2364-4ef1-9168-953b601b8278\" (UID: \"7003c4d0-2364-4ef1-9168-953b601b8278\") "
Jul 29 20:57:40.870489 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.870459    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7003c4d0-2364-4ef1-9168-953b601b8278-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7003c4d0-2364-4ef1-9168-953b601b8278" (UID: "7003c4d0-2364-4ef1-9168-953b601b8278"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:57:40.872220 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.872195    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7003c4d0-2364-4ef1-9168-953b601b8278-kube-api-access-2qjg4" (OuterVolumeSpecName: "kube-api-access-2qjg4") pod "7003c4d0-2364-4ef1-9168-953b601b8278" (UID: "7003c4d0-2364-4ef1-9168-953b601b8278"). InnerVolumeSpecName "kube-api-access-2qjg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:57:40.872320 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.872296    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7003c4d0-2364-4ef1-9168-953b601b8278-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7003c4d0-2364-4ef1-9168-953b601b8278" (UID: "7003c4d0-2364-4ef1-9168-953b601b8278"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:57:40.970856 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.970832    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7003c4d0-2364-4ef1-9168-953b601b8278-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:57:40.970856 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.970856    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7003c4d0-2364-4ef1-9168-953b601b8278-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:57:40.970992 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.970865    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qjg4\" (UniqueName: \"kubernetes.io/projected/7003c4d0-2364-4ef1-9168-953b601b8278-kube-api-access-2qjg4\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:57:40.980890 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.980867    2585 generic.go:356] "Generic (PLEG): container finished" podID="7003c4d0-2364-4ef1-9168-953b601b8278" containerID="12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0" exitCode=0
Jul 29 20:57:40.980985 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.980910    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"
Jul 29 20:57:40.980985 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.980963    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx" event={"ID":"7003c4d0-2364-4ef1-9168-953b601b8278","Type":"ContainerDied","Data":"12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0"}
Jul 29 20:57:40.981085 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.981015    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx" event={"ID":"7003c4d0-2364-4ef1-9168-953b601b8278","Type":"ContainerDied","Data":"5978600cbf1fc4aea644ff350c6d5be351345750e8138a039dbd6877317d94a6"}
Jul 29 20:57:40.981085 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.981045    2585 scope.go:117] "RemoveContainer" containerID="12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0"
Jul 29 20:57:40.989595 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.989581    2585 scope.go:117] "RemoveContainer" containerID="12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0"
Jul 29 20:57:40.989831 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:57:40.989814    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0\": container with ID starting with 12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0 not found: ID does not exist" containerID="12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0"
Jul 29 20:57:40.989877 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:40.989839    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0"} err="failed to get container status \"12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0\": rpc error: code = NotFound desc = could not find container \"12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0\": container with ID starting with 12cd902399d37a3b58963f026ae226bd423879dbc718a785fd4535b300fbbfa0 not found: ID does not exist"
Jul 29 20:57:41.005747 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:41.005722    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"]
Jul 29 20:57:41.009286 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:41.009267    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cdd0d7d-6847fb86b8-b66sx"]
Jul 29 20:57:41.525349 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:41.525294    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7003c4d0-2364-4ef1-9168-953b601b8278" path="/var/lib/kubelet/pods/7003c4d0-2364-4ef1-9168-953b601b8278/volumes"
Jul 29 20:57:41.986907 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:41.986878    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:42.215441 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:42.215408    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:42.454040 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:42.453966    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:42.778366 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:42.778314    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:42.892390 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:42.892357    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:45.043934 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.043896    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:45.145829 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.145799    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:45.293348 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.293304    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"]
Jul 29 20:57:45.293912 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.293895    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7003c4d0-2364-4ef1-9168-953b601b8278" containerName="maas-api"
Jul 29 20:57:45.294005 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.293916    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7003c4d0-2364-4ef1-9168-953b601b8278" containerName="maas-api"
Jul 29 20:57:45.294064 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.294007    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7003c4d0-2364-4ef1-9168-953b601b8278" containerName="maas-api"
Jul 29 20:57:45.298538 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.298509    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:45.301171 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.301144    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8e2d92b2-serving-cert\""
Jul 29 20:57:45.305894 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.305872    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"]
Jul 29 20:57:45.328569 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.328544    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:45.406931 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.406906    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33aed314-e893-488b-a5e9-0c582f4527cc-openshift-service-ca\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:45.407024 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.406962    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddl2v\" (UniqueName: \"kubernetes.io/projected/33aed314-e893-488b-a5e9-0c582f4527cc-kube-api-access-ddl2v\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:45.407024 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.406991    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33aed314-e893-488b-a5e9-0c582f4527cc-maas-api-tls\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:45.508233 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.508208    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddl2v\" (UniqueName: \"kubernetes.io/projected/33aed314-e893-488b-a5e9-0c582f4527cc-kube-api-access-ddl2v\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:45.508321 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.508239    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33aed314-e893-488b-a5e9-0c582f4527cc-maas-api-tls\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:45.508321 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.508293    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33aed314-e893-488b-a5e9-0c582f4527cc-openshift-service-ca\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:45.508440 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:57:45.508411    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-serving-cert: secret "maas-api-e2e-derive-8e2d92b2-serving-cert" not found
Jul 29 20:57:45.508501 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:57:45.508490    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33aed314-e893-488b-a5e9-0c582f4527cc-maas-api-tls podName:33aed314-e893-488b-a5e9-0c582f4527cc nodeName:}" failed. No retries permitted until 2026-07-29 20:57:46.008471439 +0000 UTC m=+1993.052163607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33aed314-e893-488b-a5e9-0c582f4527cc-maas-api-tls") pod "maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf" (UID: "33aed314-e893-488b-a5e9-0c582f4527cc") : secret "maas-api-e2e-derive-8e2d92b2-serving-cert" not found
Jul 29 20:57:45.508877 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.508860    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33aed314-e893-488b-a5e9-0c582f4527cc-openshift-service-ca\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:45.519370 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.519345    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddl2v\" (UniqueName: \"kubernetes.io/projected/33aed314-e893-488b-a5e9-0c582f4527cc-kube-api-access-ddl2v\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:45.616262 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.616164    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:45.799675 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.799645    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:45.881240 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:45.881175    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:46.011284 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:46.011259    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33aed314-e893-488b-a5e9-0c582f4527cc-maas-api-tls\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:46.013683 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:46.013660    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33aed314-e893-488b-a5e9-0c582f4527cc-maas-api-tls\") pod \"maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:46.211705 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:46.211629    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:46.349068 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:46.349042    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"]
Jul 29 20:57:46.351639 ip-10-0-136-123 kubenswrapper[2585]: W0729 20:57:46.351612    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33aed314_e893_488b_a5e9_0c582f4527cc.slice/crio-7bab5c228ad0153e64bc0feccb6faae422d6442c90624e375df97b4526b296ef WatchSource:0}: Error finding container 7bab5c228ad0153e64bc0feccb6faae422d6442c90624e375df97b4526b296ef: Status 404 returned error can't find the container with id 7bab5c228ad0153e64bc0feccb6faae422d6442c90624e375df97b4526b296ef
Jul 29 20:57:47.014476 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:47.014447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf" event={"ID":"33aed314-e893-488b-a5e9-0c582f4527cc","Type":"ContainerStarted","Data":"529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16"}
Jul 29 20:57:47.014622 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:47.014495    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf" event={"ID":"33aed314-e893-488b-a5e9-0c582f4527cc","Type":"ContainerStarted","Data":"7bab5c228ad0153e64bc0feccb6faae422d6442c90624e375df97b4526b296ef"}
Jul 29 20:57:47.014622 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:47.014525    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:57:47.045449 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:47.045402    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf" podStartSLOduration=1.70265672 podStartE2EDuration="2.045386535s" totalImagesPullingTime="342.729815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:57:47.043100158 +0000 UTC m=+1994.086792351" watchObservedRunningTime="2026-07-29 20:57:47.045386535 +0000 UTC m=+1994.089078717"
Jul 29 20:57:52.978358 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:52.978303    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:57:53.025231 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:57:53.025206    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:58:15.850346 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:15.850292    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"]
Jul 29 20:58:15.850839 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:15.850553    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf" podUID="33aed314-e893-488b-a5e9-0c582f4527cc" containerName="maas-api" containerID="cri-o://529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16" gracePeriod=30
Jul 29 20:58:15.878817 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:15.878795    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:15.981574 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:15.981540    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:16.079845 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.079813    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:16.105300 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.105247    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:58:16.151164 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.151137    2585 generic.go:356] "Generic (PLEG): container finished" podID="33aed314-e893-488b-a5e9-0c582f4527cc" containerID="529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16" exitCode=0
Jul 29 20:58:16.151285 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.151191    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"
Jul 29 20:58:16.151285 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.151211    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf" event={"ID":"33aed314-e893-488b-a5e9-0c582f4527cc","Type":"ContainerDied","Data":"529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16"}
Jul 29 20:58:16.151285 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.151258    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf" event={"ID":"33aed314-e893-488b-a5e9-0c582f4527cc","Type":"ContainerDied","Data":"7bab5c228ad0153e64bc0feccb6faae422d6442c90624e375df97b4526b296ef"}
Jul 29 20:58:16.151488 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.151287    2585 scope.go:117] "RemoveContainer" containerID="529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16"
Jul 29 20:58:16.151488 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.151384    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:16.153536 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.153519    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33aed314-e893-488b-a5e9-0c582f4527cc-openshift-service-ca\") pod \"33aed314-e893-488b-a5e9-0c582f4527cc\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") "
Jul 29 20:58:16.153644 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.153615    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddl2v\" (UniqueName: \"kubernetes.io/projected/33aed314-e893-488b-a5e9-0c582f4527cc-kube-api-access-ddl2v\") pod \"33aed314-e893-488b-a5e9-0c582f4527cc\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") "
Jul 29 20:58:16.153707 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.153647    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33aed314-e893-488b-a5e9-0c582f4527cc-maas-api-tls\") pod \"33aed314-e893-488b-a5e9-0c582f4527cc\" (UID: \"33aed314-e893-488b-a5e9-0c582f4527cc\") "
Jul 29 20:58:16.153955 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.153922    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33aed314-e893-488b-a5e9-0c582f4527cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33aed314-e893-488b-a5e9-0c582f4527cc" (UID: "33aed314-e893-488b-a5e9-0c582f4527cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:58:16.155615 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.155594    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33aed314-e893-488b-a5e9-0c582f4527cc-kube-api-access-ddl2v" (OuterVolumeSpecName: "kube-api-access-ddl2v") pod "33aed314-e893-488b-a5e9-0c582f4527cc" (UID: "33aed314-e893-488b-a5e9-0c582f4527cc"). InnerVolumeSpecName "kube-api-access-ddl2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:58:16.155713 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.155599    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33aed314-e893-488b-a5e9-0c582f4527cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33aed314-e893-488b-a5e9-0c582f4527cc" (UID: "33aed314-e893-488b-a5e9-0c582f4527cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:58:16.159861 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.159843    2585 scope.go:117] "RemoveContainer" containerID="529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16"
Jul 29 20:58:16.160131 ip-10-0-136-123 kubenswrapper[2585]: E0729 20:58:16.160111    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16\": container with ID starting with 529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16 not found: ID does not exist" containerID="529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16"
Jul 29 20:58:16.160177 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.160141    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16"} err="failed to get container status \"529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16\": rpc error: code = NotFound desc = could not find container \"529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16\": container with ID starting with 529260b783b961756fd799dc1396bdc0e5b9d2cc41a823f2ec83500046795f16 not found: ID does not exist"
Jul 29 20:58:16.229755 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.229727    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:16.254837 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.254815    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddl2v\" (UniqueName: \"kubernetes.io/projected/33aed314-e893-488b-a5e9-0c582f4527cc-kube-api-access-ddl2v\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:58:16.254837 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.254837    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33aed314-e893-488b-a5e9-0c582f4527cc-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:58:16.254971 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.254848    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33aed314-e893-488b-a5e9-0c582f4527cc-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 20:58:16.453690 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.453626    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:16.473443 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.473420    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"]
Jul 29 20:58:16.475080 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.475060    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e2d92b2-7b4fdd7656-cknwf"]
Jul 29 20:58:16.940549 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:16.940519    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:17.040516 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:17.040482    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:17.132732 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:17.132708    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:17.525539 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:17.525508    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33aed314-e893-488b-a5e9-0c582f4527cc" path="/var/lib/kubelet/pods/33aed314-e893-488b-a5e9-0c582f4527cc/volumes"
Jul 29 20:58:39.007670 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:39.007627    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:41.819707 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:41.819665    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:45.119945 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:45.119912    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:45.230803 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:45.230705    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:45.325449 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:45.325423    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:45.420090 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:45.419988    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:45.822159 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:45.822114    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:58:51.886319 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:58:51.886274    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:11.777905 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:11.777865    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:17.955277 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:17.955246    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:18.071606 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:18.071574    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:18.235453 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:18.235425    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:33.570841 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:33.570808    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:59:33.577609 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:33.577586    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 20:59:34.026730 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:34.026701    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:39.788828 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:39.788779    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:39.849630 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:39.849591    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:39.946842 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:39.946802    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:40.047219 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:40.047147    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 20:59:50.513454 ip-10-0-136-123 kubenswrapper[2585]: I0729 20:59:50.513417    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:01.646020 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:01.645989    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:07.520779 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:07.520748    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:07.978364 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:07.978322    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:08.050872 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:08.050820    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:08.127136 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:08.127098    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:08.540990 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:08.540955    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:11.546219 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.546150    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:11.623366 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.623320    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"]
Jul 29 21:00:11.623963 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.623944    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33aed314-e893-488b-a5e9-0c582f4527cc" containerName="maas-api"
Jul 29 21:00:11.624058 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.623965    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="33aed314-e893-488b-a5e9-0c582f4527cc" containerName="maas-api"
Jul 29 21:00:11.624058 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.624041    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="33aed314-e893-488b-a5e9-0c582f4527cc" containerName="maas-api"
Jul 29 21:00:11.627562 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.627537    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:11.630504 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.630480    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4dbbbb61-serving-cert\""
Jul 29 21:00:11.636988 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.636963    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"]
Jul 29 21:00:11.712273 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.712239    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:11.724572 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.724545    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c553b-e2eb-41e4-8efc-f18079290080-openshift-service-ca\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:11.724700 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.724627    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbvm\" (UniqueName: \"kubernetes.io/projected/ad1c553b-e2eb-41e4-8efc-f18079290080-kube-api-access-fnbvm\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:11.724700 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.724671    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1c553b-e2eb-41e4-8efc-f18079290080-maas-api-tls\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:11.816067 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.815994    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:11.825586 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.825559    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbvm\" (UniqueName: \"kubernetes.io/projected/ad1c553b-e2eb-41e4-8efc-f18079290080-kube-api-access-fnbvm\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:11.825697 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.825610    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1c553b-e2eb-41e4-8efc-f18079290080-maas-api-tls\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:11.825697 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.825659    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c553b-e2eb-41e4-8efc-f18079290080-openshift-service-ca\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:11.825813 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:00:11.825796    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-serving-cert: secret "maas-api-e2e-mt-4dbbbb61-serving-cert" not found
Jul 29 21:00:11.825889 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:00:11.825878    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1c553b-e2eb-41e4-8efc-f18079290080-maas-api-tls podName:ad1c553b-e2eb-41e4-8efc-f18079290080 nodeName:}" failed. No retries permitted until 2026-07-29 21:00:12.325858724 +0000 UTC m=+2139.369550897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad1c553b-e2eb-41e4-8efc-f18079290080-maas-api-tls") pod "maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk" (UID: "ad1c553b-e2eb-41e4-8efc-f18079290080") : secret "maas-api-e2e-mt-4dbbbb61-serving-cert" not found
Jul 29 21:00:11.826290 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.826272    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c553b-e2eb-41e4-8efc-f18079290080-openshift-service-ca\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:11.837127 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.837099    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbvm\" (UniqueName: \"kubernetes.io/projected/ad1c553b-e2eb-41e4-8efc-f18079290080-kube-api-access-fnbvm\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:11.942643 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:11.942615    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:12.207962 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:12.207883    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:12.308116 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:12.308088    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:12.331453 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:12.331414    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1c553b-e2eb-41e4-8efc-f18079290080-maas-api-tls\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:12.333770 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:12.333748    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1c553b-e2eb-41e4-8efc-f18079290080-maas-api-tls\") pod \"maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:12.540874 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:12.540844    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:12.673110 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:12.673086    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"]
Jul 29 21:00:12.675064 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:00:12.675041    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1c553b_e2eb_41e4_8efc_f18079290080.slice/crio-91ce931ead10a5916e81d931d67e9b169b596f71cdfbc23d676d32ed67796cb4 WatchSource:0}: Error finding container 91ce931ead10a5916e81d931d67e9b169b596f71cdfbc23d676d32ed67796cb4: Status 404 returned error can't find the container with id 91ce931ead10a5916e81d931d67e9b169b596f71cdfbc23d676d32ed67796cb4
Jul 29 21:00:12.709757 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:12.709732    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk" event={"ID":"ad1c553b-e2eb-41e4-8efc-f18079290080","Type":"ContainerStarted","Data":"91ce931ead10a5916e81d931d67e9b169b596f71cdfbc23d676d32ed67796cb4"}
Jul 29 21:00:13.723165 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:13.723126    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk" event={"ID":"ad1c553b-e2eb-41e4-8efc-f18079290080","Type":"ContainerStarted","Data":"7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34"}
Jul 29 21:00:13.723537 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:13.723255    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:13.739735 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:13.739696    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk" podStartSLOduration=2.096724489 podStartE2EDuration="2.739683965s" totalImagesPullingTime="642.959476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:00:13.738797518 +0000 UTC m=+2140.782489702" watchObservedRunningTime="2026-07-29 21:00:13.739683965 +0000 UTC m=+2140.783376148"
Jul 29 21:00:19.733617 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:19.733587    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:00:36.464994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:36.464882    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:37.776991 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:37.776948    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:37.879522 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:37.879484    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:48.014133 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:48.014097    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:48.386838 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:48.386753    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:48.483270 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:48.483240    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:51.768640 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.768607    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:51.859128 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.859092    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:51.860351 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.860314    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"]
Jul 29 21:00:51.863840 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.863823    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:51.867013 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.866993    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-72dabfe4-serving-cert\""
Jul 29 21:00:51.878994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.878964    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"]
Jul 29 21:00:51.949374 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.949347    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c937723-afd4-4dee-98a1-3758c795f941-maas-api-tls\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:51.949502 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.949383    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxrh\" (UniqueName: \"kubernetes.io/projected/5c937723-afd4-4dee-98a1-3758c795f941-kube-api-access-9bxrh\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:51.949502 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.949424    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c937723-afd4-4dee-98a1-3758c795f941-openshift-service-ca\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:51.952762 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:51.952739    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:52.050961 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.050878    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c937723-afd4-4dee-98a1-3758c795f941-maas-api-tls\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:52.050961 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.050926    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxrh\" (UniqueName: \"kubernetes.io/projected/5c937723-afd4-4dee-98a1-3758c795f941-kube-api-access-9bxrh\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:52.051129 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.050960    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c937723-afd4-4dee-98a1-3758c795f941-openshift-service-ca\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:52.051129 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:00:52.051041    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-serving-cert: secret "maas-api-e2e-mt-72dabfe4-serving-cert" not found
Jul 29 21:00:52.051129 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:00:52.051123    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c937723-afd4-4dee-98a1-3758c795f941-maas-api-tls podName:5c937723-afd4-4dee-98a1-3758c795f941 nodeName:}" failed. No retries permitted until 2026-07-29 21:00:52.551106458 +0000 UTC m=+2179.594798618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c937723-afd4-4dee-98a1-3758c795f941-maas-api-tls") pod "maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6" (UID: "5c937723-afd4-4dee-98a1-3758c795f941") : secret "maas-api-e2e-mt-72dabfe4-serving-cert" not found
Jul 29 21:00:52.051769 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.051749    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c937723-afd4-4dee-98a1-3758c795f941-openshift-service-ca\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:52.064223 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.064200    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxrh\" (UniqueName: \"kubernetes.io/projected/5c937723-afd4-4dee-98a1-3758c795f941-kube-api-access-9bxrh\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:52.211911 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.211877    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:52.556452 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.556422    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c937723-afd4-4dee-98a1-3758c795f941-maas-api-tls\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:52.558788 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.558764    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c937723-afd4-4dee-98a1-3758c795f941-maas-api-tls\") pod \"maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:52.778728 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.778691    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:52.919771 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.919746    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"]
Jul 29 21:00:52.921807 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:00:52.921778    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c937723_afd4_4dee_98a1_3758c795f941.slice/crio-5ffd7cc0adfaf391b96408284bcb8efaffe8eebf5652779dce20c0f7bd985c9c WatchSource:0}: Error finding container 5ffd7cc0adfaf391b96408284bcb8efaffe8eebf5652779dce20c0f7bd985c9c: Status 404 returned error can't find the container with id 5ffd7cc0adfaf391b96408284bcb8efaffe8eebf5652779dce20c0f7bd985c9c
Jul 29 21:00:52.923058 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:52.923033    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:00:53.918022 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:53.917923    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6" event={"ID":"5c937723-afd4-4dee-98a1-3758c795f941","Type":"ContainerStarted","Data":"fc1b3b159c3fced12f1834f2f33e6cc0a96de278895a0cc11bf10a198ebbe9fc"}
Jul 29 21:00:53.918022 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:53.917988    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6" event={"ID":"5c937723-afd4-4dee-98a1-3758c795f941","Type":"ContainerStarted","Data":"5ffd7cc0adfaf391b96408284bcb8efaffe8eebf5652779dce20c0f7bd985c9c"}
Jul 29 21:00:53.918022 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:53.918018    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:00:53.926264 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:53.926236    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:53.938272 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:53.938230    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6" podStartSLOduration=2.381978678 podStartE2EDuration="2.938216878s" totalImagesPullingTime="556.2382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:00:53.935762803 +0000 UTC m=+2180.979454985" watchObservedRunningTime="2026-07-29 21:00:53.938216878 +0000 UTC m=+2180.981909061"
Jul 29 21:00:54.028227 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:54.028200    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:00:59.927677 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:00:59.927644    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:01:21.864409 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:01:21.864368    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:01:53.521702 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:01:53.521662    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:01:56.537774 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:01:56.537687    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:04:33.608476 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:04:33.608447    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:04:33.614848 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:04:33.614826    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:05:56.665249 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:05:56.665214    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:06:15.215980 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:06:15.215945    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:06:27.414405 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:06:27.414301    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:08:08.258084 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:08:08.258005    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:08:14.390445 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:08:14.390412    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:09:33.639345 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:33.639260    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:09:33.646418 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:33.646389    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:09:38.520678 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:38.520649    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:09:38.740071 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:38.740040    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:09:54.393883 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.393846    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"]
Jul 29 21:09:54.394301 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.394047    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6" podUID="5c937723-afd4-4dee-98a1-3758c795f941" containerName="maas-api" containerID="cri-o://fc1b3b159c3fced12f1834f2f33e6cc0a96de278895a0cc11bf10a198ebbe9fc" gracePeriod=30
Jul 29 21:09:54.422224 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.422198    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:09:54.513705 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.513670    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:09:54.521858 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.521832    2585 generic.go:356] "Generic (PLEG): container finished" podID="5c937723-afd4-4dee-98a1-3758c795f941" containerID="fc1b3b159c3fced12f1834f2f33e6cc0a96de278895a0cc11bf10a198ebbe9fc" exitCode=0
Jul 29 21:09:54.521973 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.521905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6" event={"ID":"5c937723-afd4-4dee-98a1-3758c795f941","Type":"ContainerDied","Data":"fc1b3b159c3fced12f1834f2f33e6cc0a96de278895a0cc11bf10a198ebbe9fc"}
Jul 29 21:09:54.613646 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.613606    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:09:54.646367 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.646292    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:09:54.713267 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.713242    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c937723-afd4-4dee-98a1-3758c795f941-openshift-service-ca\") pod \"5c937723-afd4-4dee-98a1-3758c795f941\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") "
Jul 29 21:09:54.713409 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.713321    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bxrh\" (UniqueName: \"kubernetes.io/projected/5c937723-afd4-4dee-98a1-3758c795f941-kube-api-access-9bxrh\") pod \"5c937723-afd4-4dee-98a1-3758c795f941\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") "
Jul 29 21:09:54.713409 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.713404    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c937723-afd4-4dee-98a1-3758c795f941-maas-api-tls\") pod \"5c937723-afd4-4dee-98a1-3758c795f941\" (UID: \"5c937723-afd4-4dee-98a1-3758c795f941\") "
Jul 29 21:09:54.713626 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.713605    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c937723-afd4-4dee-98a1-3758c795f941-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c937723-afd4-4dee-98a1-3758c795f941" (UID: "5c937723-afd4-4dee-98a1-3758c795f941"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:09:54.715308 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.715291    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c937723-afd4-4dee-98a1-3758c795f941-kube-api-access-9bxrh" (OuterVolumeSpecName: "kube-api-access-9bxrh") pod "5c937723-afd4-4dee-98a1-3758c795f941" (UID: "5c937723-afd4-4dee-98a1-3758c795f941"). InnerVolumeSpecName "kube-api-access-9bxrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:09:54.715473 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.715453    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c937723-afd4-4dee-98a1-3758c795f941-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c937723-afd4-4dee-98a1-3758c795f941" (UID: "5c937723-afd4-4dee-98a1-3758c795f941"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:09:54.814852 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.814821    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c937723-afd4-4dee-98a1-3758c795f941-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:09:54.814852 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.814845    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bxrh\" (UniqueName: \"kubernetes.io/projected/5c937723-afd4-4dee-98a1-3758c795f941-kube-api-access-9bxrh\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:09:54.814852 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:54.814855    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c937723-afd4-4dee-98a1-3758c795f941-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:09:55.001657 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:55.001624    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:09:55.527813 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:55.527784    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"
Jul 29 21:09:55.528264 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:55.527810    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6" event={"ID":"5c937723-afd4-4dee-98a1-3758c795f941","Type":"ContainerDied","Data":"5ffd7cc0adfaf391b96408284bcb8efaffe8eebf5652779dce20c0f7bd985c9c"}
Jul 29 21:09:55.528264 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:55.527864    2585 scope.go:117] "RemoveContainer" containerID="fc1b3b159c3fced12f1834f2f33e6cc0a96de278895a0cc11bf10a198ebbe9fc"
Jul 29 21:09:55.556899 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:55.556873    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"]
Jul 29 21:09:55.558543 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:55.558523    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-72dabfe4-5d44f449c5-4cjj6"]
Jul 29 21:09:57.524697 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:09:57.524658    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c937723-afd4-4dee-98a1-3758c795f941" path="/var/lib/kubelet/pods/5c937723-afd4-4dee-98a1-3758c795f941/volumes"
Jul 29 21:10:00.760734 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:00.760699    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:00.825177 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:00.825142    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:01.034639 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:01.034557    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:01.254424 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:01.254386    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:01.542226 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:01.542195    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:01.926605 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:01.926519    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:02.384998 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:02.384964    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:04.733203 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:04.733170    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:04.854492 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:04.854453    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:05.549080 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.549048    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"]
Jul 29 21:10:05.549522 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.549503    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c937723-afd4-4dee-98a1-3758c795f941" containerName="maas-api"
Jul 29 21:10:05.549636 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.549523    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c937723-afd4-4dee-98a1-3758c795f941" containerName="maas-api"
Jul 29 21:10:05.549636 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.549629    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c937723-afd4-4dee-98a1-3758c795f941" containerName="maas-api"
Jul 29 21:10:05.554123 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.554101    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.556878 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.556856    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0ab55cec-serving-cert\""
Jul 29 21:10:05.571713 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.571688    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"]
Jul 29 21:10:05.604341 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.604302    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-openshift-service-ca\") pod \"maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.604462 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.604427    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmmbv\" (UniqueName: \"kubernetes.io/projected/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-kube-api-access-fmmbv\") pod \"maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.604533 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.604490    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-maas-api-tls\") pod \"maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.606812 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.606789    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:05.705031 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.704999    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-openshift-service-ca\") pod \"maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.705192 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.705054    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmmbv\" (UniqueName: \"kubernetes.io/projected/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-kube-api-access-fmmbv\") pod \"maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.705192 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.705093    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-maas-api-tls\") pod \"maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.705703 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.705678    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-openshift-service-ca\") pod \"maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.707547 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.707527    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-maas-api-tls\") pod \"maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.713234 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.713210    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmmbv\" (UniqueName: \"kubernetes.io/projected/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-kube-api-access-fmmbv\") pod \"maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:05.751627 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.751594    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:05.863940 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:05.863853    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:06.003047 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:06.003019    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"]
Jul 29 21:10:06.004986 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:10:06.004952    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c474a7_fe08_43f1_bcc9_8d529fc469c7.slice/crio-2726c46d4558eeff385839539336715380686febc2a116d0f7726328f5ae939e WatchSource:0}: Error finding container 2726c46d4558eeff385839539336715380686febc2a116d0f7726328f5ae939e: Status 404 returned error can't find the container with id 2726c46d4558eeff385839539336715380686febc2a116d0f7726328f5ae939e
Jul 29 21:10:06.006516 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:06.006495    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:10:06.131491 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:06.131397    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:06.242965 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:06.242939    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:06.581269 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:06.581241    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf" event={"ID":"f6c474a7-fe08-43f1-bcc9-8d529fc469c7","Type":"ContainerStarted","Data":"9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90"}
Jul 29 21:10:06.581394 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:06.581288    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf" event={"ID":"f6c474a7-fe08-43f1-bcc9-8d529fc469c7","Type":"ContainerStarted","Data":"2726c46d4558eeff385839539336715380686febc2a116d0f7726328f5ae939e"}
Jul 29 21:10:06.581448 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:06.581387    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:06.597693 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:06.597642    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf" podStartSLOduration=1.087464982 podStartE2EDuration="1.597626778s" totalImagesPullingTime="510.161796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:10:06.596418477 +0000 UTC m=+2733.640110660" watchObservedRunningTime="2026-07-29 21:10:06.597626778 +0000 UTC m=+2733.641318961"
Jul 29 21:10:12.592060 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:12.592030    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:14.223494 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:14.223456    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:15.000818 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:15.000785    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:15.132951 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:15.132919    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:15.336632 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:15.336531    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:21.325063 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:21.325023    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:21.424365 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:21.424318    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:27.346443 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.346405    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"]
Jul 29 21:10:27.346835 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.346653    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf" podUID="f6c474a7-fe08-43f1-bcc9-8d529fc469c7" containerName="maas-api" containerID="cri-o://9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90" gracePeriod=30
Jul 29 21:10:27.373111 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.373071    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:27.475392 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.475317    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:27.570067 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.570033    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:27.592963 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.592945    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:27.631946 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.631878    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:27.686632 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.686604    2585 generic.go:356] "Generic (PLEG): container finished" podID="f6c474a7-fe08-43f1-bcc9-8d529fc469c7" containerID="9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90" exitCode=0
Jul 29 21:10:27.686810 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.686674    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"
Jul 29 21:10:27.686810 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.686651    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf" event={"ID":"f6c474a7-fe08-43f1-bcc9-8d529fc469c7","Type":"ContainerDied","Data":"9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90"}
Jul 29 21:10:27.686923 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.686832    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf" event={"ID":"f6c474a7-fe08-43f1-bcc9-8d529fc469c7","Type":"ContainerDied","Data":"2726c46d4558eeff385839539336715380686febc2a116d0f7726328f5ae939e"}
Jul 29 21:10:27.686923 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.686877    2585 scope.go:117] "RemoveContainer" containerID="9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90"
Jul 29 21:10:27.694593 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.694569    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmmbv\" (UniqueName: \"kubernetes.io/projected/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-kube-api-access-fmmbv\") pod \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") "
Jul 29 21:10:27.694718 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.694640    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-maas-api-tls\") pod \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") "
Jul 29 21:10:27.694808 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.694793    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-openshift-service-ca\") pod \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\" (UID: \"f6c474a7-fe08-43f1-bcc9-8d529fc469c7\") "
Jul 29 21:10:27.695178 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.695149    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6c474a7-fe08-43f1-bcc9-8d529fc469c7" (UID: "f6c474a7-fe08-43f1-bcc9-8d529fc469c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:10:27.696587 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.696563    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6c474a7-fe08-43f1-bcc9-8d529fc469c7" (UID: "f6c474a7-fe08-43f1-bcc9-8d529fc469c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:10:27.696687 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.696668    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-kube-api-access-fmmbv" (OuterVolumeSpecName: "kube-api-access-fmmbv") pod "f6c474a7-fe08-43f1-bcc9-8d529fc469c7" (UID: "f6c474a7-fe08-43f1-bcc9-8d529fc469c7"). InnerVolumeSpecName "kube-api-access-fmmbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:10:27.697128 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.697109    2585 scope.go:117] "RemoveContainer" containerID="9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90"
Jul 29 21:10:27.697489 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:10:27.697457    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90\": container with ID starting with 9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90 not found: ID does not exist" containerID="9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90"
Jul 29 21:10:27.697579 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.697499    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90"} err="failed to get container status \"9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90\": rpc error: code = NotFound desc = could not find container \"9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90\": container with ID starting with 9c291fe276ae68a11b4ef39d4415297ec885800596569666766e5f2caa75ce90 not found: ID does not exist"
Jul 29 21:10:27.796300 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.796278    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:10:27.796414 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.796303    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmmbv\" (UniqueName: \"kubernetes.io/projected/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-kube-api-access-fmmbv\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:10:27.796414 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.796317    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c474a7-fe08-43f1-bcc9-8d529fc469c7-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:10:27.923576 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:27.923526    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:28.009035 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:28.009002    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"]
Jul 29 21:10:28.012475 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:28.012447    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf"]
Jul 29 21:10:29.524610 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:29.524565    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c474a7-fe08-43f1-bcc9-8d529fc469c7" path="/var/lib/kubelet/pods/f6c474a7-fe08-43f1-bcc9-8d529fc469c7/volumes"
Jul 29 21:10:30.588278 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:30.588235    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0ab55cec-787744b4fd-hdzgf" podUID="f6c474a7-fe08-43f1-bcc9-8d529fc469c7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.48:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 29 21:10:33.618905 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:33.618865    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:33.684884 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:33.684860    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:33.921954 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:33.921869    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:34.133318 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:34.133288    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:34.250552 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:34.250523    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:34.352275 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:34.352253    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:34.471534 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:34.471505    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:40.387572 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:40.387535    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:40.489147 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:40.489117    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:40.637016 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:40.636975    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:40.737602 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:40.737568    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:46.236139 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:46.236100    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:10:46.305178 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:10:46.305146    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:11:21.212987 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:11:21.212946    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:11:32.832062 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:11:32.832030    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:11:32.881489 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:11:32.881465    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:11:53.733099 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:11:53.733056    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:12:59.170913 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:12:59.170876    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:12:59.461835 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:12:59.461755    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:12:59.910522 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:12:59.910490    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:13:02.696503 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.696455    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:13:02.761173 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.761143    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"]
Jul 29 21:13:02.761692 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.761667    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c474a7-fe08-43f1-bcc9-8d529fc469c7" containerName="maas-api"
Jul 29 21:13:02.761692 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.761692    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c474a7-fe08-43f1-bcc9-8d529fc469c7" containerName="maas-api"
Jul 29 21:13:02.761973 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.761790    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c474a7-fe08-43f1-bcc9-8d529fc469c7" containerName="maas-api"
Jul 29 21:13:02.764908 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.764884    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:02.767532 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.767510    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6951c5fd-serving-cert\""
Jul 29 21:13:02.767701 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.767670    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:13:02.777264 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.777215    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"]
Jul 29 21:13:02.841906 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.841858    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19abcc0a-506f-420c-b2a8-5766a0dd528b-maas-api-tls\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:02.842046 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.841951    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19abcc0a-506f-420c-b2a8-5766a0dd528b-openshift-service-ca\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:02.842046 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.841991    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c849\" (UniqueName: \"kubernetes.io/projected/19abcc0a-506f-420c-b2a8-5766a0dd528b-kube-api-access-6c849\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:02.842570 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.842544    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:13:02.942928 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.942874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19abcc0a-506f-420c-b2a8-5766a0dd528b-openshift-service-ca\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:02.942928 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.942902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c849\" (UniqueName: \"kubernetes.io/projected/19abcc0a-506f-420c-b2a8-5766a0dd528b-kube-api-access-6c849\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:02.943098 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.942956    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19abcc0a-506f-420c-b2a8-5766a0dd528b-maas-api-tls\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:02.943098 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:13:02.943076    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-serving-cert: secret "maas-api-e2e-mt-6951c5fd-serving-cert" not found
Jul 29 21:13:02.943211 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:13:02.943132    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19abcc0a-506f-420c-b2a8-5766a0dd528b-maas-api-tls podName:19abcc0a-506f-420c-b2a8-5766a0dd528b nodeName:}" failed. No retries permitted until 2026-07-29 21:13:03.443115673 +0000 UTC m=+2910.486807849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19abcc0a-506f-420c-b2a8-5766a0dd528b-maas-api-tls") pod "maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf" (UID: "19abcc0a-506f-420c-b2a8-5766a0dd528b") : secret "maas-api-e2e-mt-6951c5fd-serving-cert" not found
Jul 29 21:13:02.943585 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.943567    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19abcc0a-506f-420c-b2a8-5766a0dd528b-openshift-service-ca\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:02.955557 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:02.955530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c849\" (UniqueName: \"kubernetes.io/projected/19abcc0a-506f-420c-b2a8-5766a0dd528b-kube-api-access-6c849\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:03.061405 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:03.061385    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:13:03.340478 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:03.340449    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:13:03.444868 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:03.444840    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:13:03.447990 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:03.447968    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19abcc0a-506f-420c-b2a8-5766a0dd528b-maas-api-tls\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:03.450308 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:03.450278    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19abcc0a-506f-420c-b2a8-5766a0dd528b-maas-api-tls\") pod \"maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:03.678202 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:03.678106    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:03.809642 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:03.809619    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"]
Jul 29 21:13:03.811641 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:13:03.811606    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19abcc0a_506f_420c_b2a8_5766a0dd528b.slice/crio-c13f87de80f947fee515f357fd0a96ad3f1e1dba27e2718baa765fbb8d733044 WatchSource:0}: Error finding container c13f87de80f947fee515f357fd0a96ad3f1e1dba27e2718baa765fbb8d733044: Status 404 returned error can't find the container with id c13f87de80f947fee515f357fd0a96ad3f1e1dba27e2718baa765fbb8d733044
Jul 29 21:13:04.435717 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:04.435692    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf" event={"ID":"19abcc0a-506f-420c-b2a8-5766a0dd528b","Type":"ContainerStarted","Data":"442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425"}
Jul 29 21:13:04.435828 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:04.435738    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf" event={"ID":"19abcc0a-506f-420c-b2a8-5766a0dd528b","Type":"ContainerStarted","Data":"c13f87de80f947fee515f357fd0a96ad3f1e1dba27e2718baa765fbb8d733044"}
Jul 29 21:13:04.435828 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:04.435765    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:04.455184 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:04.455143    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf" podStartSLOduration=1.8978468450000001 podStartE2EDuration="2.455123227s" totalImagesPullingTime="557.276382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:13:04.454105963 +0000 UTC m=+2911.497798160" watchObservedRunningTime="2026-07-29 21:13:04.455123227 +0000 UTC m=+2911.498815409"
Jul 29 21:13:10.445804 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:10.445773    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:13:27.689136 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:27.689096    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:13:31.887507 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:13:31.887474    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:22.772083 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:22.772047    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"]
Jul 29 21:14:22.772552 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:22.772243    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf" podUID="19abcc0a-506f-420c-b2a8-5766a0dd528b" containerName="maas-api" containerID="cri-o://442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425" gracePeriod=30
Jul 29 21:14:22.791938 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:22.791906    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:22.889444 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:22.889395    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:22.979437 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:22.979403    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:23.020481 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.020454    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:14:23.044823 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.044757    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:23.084449 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.084421    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19abcc0a-506f-420c-b2a8-5766a0dd528b-openshift-service-ca\") pod \"19abcc0a-506f-420c-b2a8-5766a0dd528b\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") "
Jul 29 21:14:23.084581 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.084459    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c849\" (UniqueName: \"kubernetes.io/projected/19abcc0a-506f-420c-b2a8-5766a0dd528b-kube-api-access-6c849\") pod \"19abcc0a-506f-420c-b2a8-5766a0dd528b\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") "
Jul 29 21:14:23.084581 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.084565    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19abcc0a-506f-420c-b2a8-5766a0dd528b-maas-api-tls\") pod \"19abcc0a-506f-420c-b2a8-5766a0dd528b\" (UID: \"19abcc0a-506f-420c-b2a8-5766a0dd528b\") "
Jul 29 21:14:23.084797 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.084774    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19abcc0a-506f-420c-b2a8-5766a0dd528b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19abcc0a-506f-420c-b2a8-5766a0dd528b" (UID: "19abcc0a-506f-420c-b2a8-5766a0dd528b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:14:23.084921 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.084899    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19abcc0a-506f-420c-b2a8-5766a0dd528b-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:14:23.086544 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.086523    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19abcc0a-506f-420c-b2a8-5766a0dd528b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19abcc0a-506f-420c-b2a8-5766a0dd528b" (UID: "19abcc0a-506f-420c-b2a8-5766a0dd528b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:14:23.086622 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.086565    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19abcc0a-506f-420c-b2a8-5766a0dd528b-kube-api-access-6c849" (OuterVolumeSpecName: "kube-api-access-6c849") pod "19abcc0a-506f-420c-b2a8-5766a0dd528b" (UID: "19abcc0a-506f-420c-b2a8-5766a0dd528b"). InnerVolumeSpecName "kube-api-access-6c849". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:14:23.186369 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.186319    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19abcc0a-506f-420c-b2a8-5766a0dd528b-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:14:23.186369 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.186365    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c849\" (UniqueName: \"kubernetes.io/projected/19abcc0a-506f-420c-b2a8-5766a0dd528b-kube-api-access-6c849\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:14:23.817543 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.817453    2585 generic.go:356] "Generic (PLEG): container finished" podID="19abcc0a-506f-420c-b2a8-5766a0dd528b" containerID="442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425" exitCode=0
Jul 29 21:14:23.817543 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.817498    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"
Jul 29 21:14:23.817543 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.817525    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf" event={"ID":"19abcc0a-506f-420c-b2a8-5766a0dd528b","Type":"ContainerDied","Data":"442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425"}
Jul 29 21:14:23.818075 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.817572    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf" event={"ID":"19abcc0a-506f-420c-b2a8-5766a0dd528b","Type":"ContainerDied","Data":"c13f87de80f947fee515f357fd0a96ad3f1e1dba27e2718baa765fbb8d733044"}
Jul 29 21:14:23.818075 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.817594    2585 scope.go:117] "RemoveContainer" containerID="442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425"
Jul 29 21:14:23.826392 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.826364    2585 scope.go:117] "RemoveContainer" containerID="442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425"
Jul 29 21:14:23.826670 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:14:23.826651    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425\": container with ID starting with 442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425 not found: ID does not exist" containerID="442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425"
Jul 29 21:14:23.826753 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.826679    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425"} err="failed to get container status \"442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425\": rpc error: code = NotFound desc = could not find container \"442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425\": container with ID starting with 442c9720679443e721fe69f060b71195033b5881786ea824d609505521b01425 not found: ID does not exist"
Jul 29 21:14:23.842016 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.841993    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"]
Jul 29 21:14:23.846277 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:23.846258    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6951c5fd-67644ffd65-l6srf"]
Jul 29 21:14:25.524516 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:25.524483    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19abcc0a-506f-420c-b2a8-5766a0dd528b" path="/var/lib/kubelet/pods/19abcc0a-506f-420c-b2a8-5766a0dd528b/volumes"
Jul 29 21:14:26.242388 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:26.242351    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:31.321096 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:31.321062    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:33.672266 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:33.672236    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:14:33.677555 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:33.677532    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:14:38.461575 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.461534    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"]
Jul 29 21:14:38.461989 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.461795    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk" podUID="ad1c553b-e2eb-41e4-8efc-f18079290080" containerName="maas-api" containerID="cri-o://7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34" gracePeriod=30
Jul 29 21:14:38.491858 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.491822    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:38.593755 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.593719    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:38.656064 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.656028    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:38.708053 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.708032    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:14:38.812485 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.812453    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c553b-e2eb-41e4-8efc-f18079290080-openshift-service-ca\") pod \"ad1c553b-e2eb-41e4-8efc-f18079290080\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") "
Jul 29 21:14:38.812626 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.812511    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1c553b-e2eb-41e4-8efc-f18079290080-maas-api-tls\") pod \"ad1c553b-e2eb-41e4-8efc-f18079290080\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") "
Jul 29 21:14:38.812626 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.812535    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnbvm\" (UniqueName: \"kubernetes.io/projected/ad1c553b-e2eb-41e4-8efc-f18079290080-kube-api-access-fnbvm\") pod \"ad1c553b-e2eb-41e4-8efc-f18079290080\" (UID: \"ad1c553b-e2eb-41e4-8efc-f18079290080\") "
Jul 29 21:14:38.812796 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.812773    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1c553b-e2eb-41e4-8efc-f18079290080-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad1c553b-e2eb-41e4-8efc-f18079290080" (UID: "ad1c553b-e2eb-41e4-8efc-f18079290080"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:14:38.814513 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.814485    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1c553b-e2eb-41e4-8efc-f18079290080-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad1c553b-e2eb-41e4-8efc-f18079290080" (UID: "ad1c553b-e2eb-41e4-8efc-f18079290080"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:14:38.814643 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.814624    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1c553b-e2eb-41e4-8efc-f18079290080-kube-api-access-fnbvm" (OuterVolumeSpecName: "kube-api-access-fnbvm") pod "ad1c553b-e2eb-41e4-8efc-f18079290080" (UID: "ad1c553b-e2eb-41e4-8efc-f18079290080"). InnerVolumeSpecName "kube-api-access-fnbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:14:38.896408 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.896379    2585 generic.go:356] "Generic (PLEG): container finished" podID="ad1c553b-e2eb-41e4-8efc-f18079290080" containerID="7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34" exitCode=0
Jul 29 21:14:38.896516 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.896431    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"
Jul 29 21:14:38.896516 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.896467    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk" event={"ID":"ad1c553b-e2eb-41e4-8efc-f18079290080","Type":"ContainerDied","Data":"7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34"}
Jul 29 21:14:38.896628 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.896520    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk" event={"ID":"ad1c553b-e2eb-41e4-8efc-f18079290080","Type":"ContainerDied","Data":"91ce931ead10a5916e81d931d67e9b169b596f71cdfbc23d676d32ed67796cb4"}
Jul 29 21:14:38.896628 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.896543    2585 scope.go:117] "RemoveContainer" containerID="7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34"
Jul 29 21:14:38.905819 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.905805    2585 scope.go:117] "RemoveContainer" containerID="7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34"
Jul 29 21:14:38.906062 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:14:38.906044    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34\": container with ID starting with 7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34 not found: ID does not exist" containerID="7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34"
Jul 29 21:14:38.906113 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.906069    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34"} err="failed to get container status \"7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34\": rpc error: code = NotFound desc = could not find container \"7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34\": container with ID starting with 7f3681a0c715ef1c268cf2370d54b1ccdadd40eac3db4edafcc23a1fe6bdbc34 not found: ID does not exist"
Jul 29 21:14:38.913738 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.913714    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c553b-e2eb-41e4-8efc-f18079290080-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:14:38.913822 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.913739    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1c553b-e2eb-41e4-8efc-f18079290080-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:14:38.913822 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.913755    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnbvm\" (UniqueName: \"kubernetes.io/projected/ad1c553b-e2eb-41e4-8efc-f18079290080-kube-api-access-fnbvm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:14:38.918485 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.918461    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"]
Jul 29 21:14:38.921773 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:38.921749    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4dbbbb61-bc554d6f-rg8gk"]
Jul 29 21:14:39.524637 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:39.524592    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad1c553b-e2eb-41e4-8efc-f18079290080" path="/var/lib/kubelet/pods/ad1c553b-e2eb-41e4-8efc-f18079290080/volumes"
Jul 29 21:14:41.865215 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:41.865165    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:14:41.965002 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:14:41.964965    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:07.480179 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:07.480143    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:08.279137 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:08.279098    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:13.842564 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:13.842524    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:13.942740 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:13.942705    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:14.166461 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:14.166377    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:14.365794 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:14.365759    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:36.082653 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:36.082603    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:36.405004 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:36.404929    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:36.607699 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:36.607668    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:39.578537 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.578499    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:39.685009 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.684975    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:39.686534 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.686510    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"]
Jul 29 21:15:39.686866 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.686853    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19abcc0a-506f-420c-b2a8-5766a0dd528b" containerName="maas-api"
Jul 29 21:15:39.686915 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.686867    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="19abcc0a-506f-420c-b2a8-5766a0dd528b" containerName="maas-api"
Jul 29 21:15:39.686915 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.686894    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad1c553b-e2eb-41e4-8efc-f18079290080" containerName="maas-api"
Jul 29 21:15:39.686915 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.686900    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad1c553b-e2eb-41e4-8efc-f18079290080" containerName="maas-api"
Jul 29 21:15:39.687010 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.686951    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad1c553b-e2eb-41e4-8efc-f18079290080" containerName="maas-api"
Jul 29 21:15:39.687010 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.686961    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="19abcc0a-506f-420c-b2a8-5766a0dd528b" containerName="maas-api"
Jul 29 21:15:39.691165 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.691139    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:39.694039 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.694021    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fcb1aff7-serving-cert\""
Jul 29 21:15:39.701369 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.701348    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"]
Jul 29 21:15:39.774763 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.774737    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:39.823625 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.823602    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be190b1d-ad17-4a0e-8cfd-aacefbff6551-maas-api-tls\") pod \"maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:39.823744 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.823646    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxcq\" (UniqueName: \"kubernetes.io/projected/be190b1d-ad17-4a0e-8cfd-aacefbff6551-kube-api-access-wqxcq\") pod \"maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:39.823744 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.823719    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be190b1d-ad17-4a0e-8cfd-aacefbff6551-openshift-service-ca\") pod \"maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:39.876541 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.876475    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:39.924591 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.924568    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be190b1d-ad17-4a0e-8cfd-aacefbff6551-maas-api-tls\") pod \"maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:39.924685 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.924599    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxcq\" (UniqueName: \"kubernetes.io/projected/be190b1d-ad17-4a0e-8cfd-aacefbff6551-kube-api-access-wqxcq\") pod \"maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:39.924685 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.924631    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be190b1d-ad17-4a0e-8cfd-aacefbff6551-openshift-service-ca\") pod \"maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:39.925153 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.925137    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be190b1d-ad17-4a0e-8cfd-aacefbff6551-openshift-service-ca\") pod \"maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:39.926977 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.926959    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be190b1d-ad17-4a0e-8cfd-aacefbff6551-maas-api-tls\") pod \"maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:39.934082 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:39.934059    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxcq\" (UniqueName: \"kubernetes.io/projected/be190b1d-ad17-4a0e-8cfd-aacefbff6551-kube-api-access-wqxcq\") pod \"maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:40.003865 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:40.003846    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:40.147249 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:40.147168    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"]
Jul 29 21:15:40.149828 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:15:40.149797    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe190b1d_ad17_4a0e_8cfd_aacefbff6551.slice/crio-27197c7de7966333e52fa3a37af6d825cfdde0450d2d306e19f79f086b524f70 WatchSource:0}: Error finding container 27197c7de7966333e52fa3a37af6d825cfdde0450d2d306e19f79f086b524f70: Status 404 returned error can't find the container with id 27197c7de7966333e52fa3a37af6d825cfdde0450d2d306e19f79f086b524f70
Jul 29 21:15:40.151242 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:40.151224    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:15:40.204092 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:40.204056    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7" event={"ID":"be190b1d-ad17-4a0e-8cfd-aacefbff6551","Type":"ContainerStarted","Data":"27197c7de7966333e52fa3a37af6d825cfdde0450d2d306e19f79f086b524f70"}
Jul 29 21:15:40.315061 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:40.315027    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:40.419375 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:40.419278    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:41.210808 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:41.210718    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7" event={"ID":"be190b1d-ad17-4a0e-8cfd-aacefbff6551","Type":"ContainerStarted","Data":"5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a"}
Jul 29 21:15:41.211139 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:41.210819    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:15:41.229292 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:41.229248    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7" podStartSLOduration=1.626653944 podStartE2EDuration="2.229234282s" totalImagesPullingTime="602.580338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:15:41.22691435 +0000 UTC m=+3068.270606534" watchObservedRunningTime="2026-07-29 21:15:41.229234282 +0000 UTC m=+3068.272926465"
Jul 29 21:15:42.261696 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:42.261665    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:42.311183 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:42.311143    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:15:47.220352 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:15:47.220304    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:16:04.619486 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:04.619449    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:15.791317 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:15.791273    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:16.104229 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:16.104136    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:16.465522 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:16.465488    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:18.832634 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:18.832591    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:18.916078 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:18.916041    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"]
Jul 29 21:16:18.919789 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:18.919767    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:18.922385 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:18.922362    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-aa96fae3-serving-cert\""
Jul 29 21:16:18.934314 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:18.934290    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"]
Jul 29 21:16:18.950438 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:18.950415    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67234690-8c0f-4bc7-8e4d-abd82315ce9a-openshift-service-ca\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:18.950554 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:18.950449    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67234690-8c0f-4bc7-8e4d-abd82315ce9a-maas-api-tls\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:18.950554 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:18.950485    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvwzz\" (UniqueName: \"kubernetes.io/projected/67234690-8c0f-4bc7-8e4d-abd82315ce9a-kube-api-access-dvwzz\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:18.962862 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:18.962835    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:19.051165 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.051142    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvwzz\" (UniqueName: \"kubernetes.io/projected/67234690-8c0f-4bc7-8e4d-abd82315ce9a-kube-api-access-dvwzz\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:19.051258 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.051207    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67234690-8c0f-4bc7-8e4d-abd82315ce9a-openshift-service-ca\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:19.051258 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.051229    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67234690-8c0f-4bc7-8e4d-abd82315ce9a-maas-api-tls\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:19.051398 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:16:19.051316    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-serving-cert: secret "maas-api-e2e-mt-aa96fae3-serving-cert" not found
Jul 29 21:16:19.051398 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:16:19.051399    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67234690-8c0f-4bc7-8e4d-abd82315ce9a-maas-api-tls podName:67234690-8c0f-4bc7-8e4d-abd82315ce9a nodeName:}" failed. No retries permitted until 2026-07-29 21:16:19.551384589 +0000 UTC m=+3106.595076749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67234690-8c0f-4bc7-8e4d-abd82315ce9a-maas-api-tls") pod "maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc" (UID: "67234690-8c0f-4bc7-8e4d-abd82315ce9a") : secret "maas-api-e2e-mt-aa96fae3-serving-cert" not found
Jul 29 21:16:19.051903 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.051880    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67234690-8c0f-4bc7-8e4d-abd82315ce9a-openshift-service-ca\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:19.062121 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.062094    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvwzz\" (UniqueName: \"kubernetes.io/projected/67234690-8c0f-4bc7-8e4d-abd82315ce9a-kube-api-access-dvwzz\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:19.065087 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.065059    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:19.206441 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.206384    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:19.555083 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.555048    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67234690-8c0f-4bc7-8e4d-abd82315ce9a-maas-api-tls\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:19.557484 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.557460    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67234690-8c0f-4bc7-8e4d-abd82315ce9a-maas-api-tls\") pod \"maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:19.560673 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.560646    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:19.658900 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.658872    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:16:19.832816 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.832743    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:19.966056 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:19.966025    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"]
Jul 29 21:16:19.968182 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:16:19.968158    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67234690_8c0f_4bc7_8e4d_abd82315ce9a.slice/crio-ace98c2dc069c412ffa33576b461522cc582c86cbeda100c6bc1cb3733c60614 WatchSource:0}: Error finding container ace98c2dc069c412ffa33576b461522cc582c86cbeda100c6bc1cb3733c60614: Status 404 returned error can't find the container with id ace98c2dc069c412ffa33576b461522cc582c86cbeda100c6bc1cb3733c60614
Jul 29 21:16:20.401662 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:20.401633    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc" event={"ID":"67234690-8c0f-4bc7-8e4d-abd82315ce9a","Type":"ContainerStarted","Data":"ace98c2dc069c412ffa33576b461522cc582c86cbeda100c6bc1cb3733c60614"}
Jul 29 21:16:21.408235 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:21.408199    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc" event={"ID":"67234690-8c0f-4bc7-8e4d-abd82315ce9a","Type":"ContainerStarted","Data":"6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e"}
Jul 29 21:16:21.410465 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:21.408277    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:21.428873 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:21.428793    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc" podStartSLOduration=3.016545108 podStartE2EDuration="3.428772948s" totalImagesPullingTime="412.22784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:21.425135113 +0000 UTC m=+3108.468827298" watchObservedRunningTime="2026-07-29 21:16:21.428772948 +0000 UTC m=+3108.472465135"
Jul 29 21:16:27.420199 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:27.420159    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:16:43.869633 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:16:43.869594    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:17:18.549038 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:17:18.548998    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:17:27.604259 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:17:27.604217    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:17:27.658014 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:17:27.657985    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:18:08.073498 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:18:08.073462    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:18:23.913245 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:18:23.913207    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:18:58.070578 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:18:58.070536    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:19:33.713893 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:19:33.713863    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:19:33.717211 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:19:33.717187    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:21:41.490440 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:21:41.490343    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:23:30.216244 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:23:30.216165    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:23:35.849799 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:23:35.849762    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:24:33.744964 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:24:33.744932    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:24:33.749487 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:24:33.749459    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:25:00.319911 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:00.319879    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:02.858955 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:02.858925    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:16.527528 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.527492    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"]
Jul 29 21:25:16.527998 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.527759    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc" podUID="67234690-8c0f-4bc7-8e4d-abd82315ce9a" containerName="maas-api" containerID="cri-o://6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e" gracePeriod=30
Jul 29 21:25:16.556094 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.556066    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:16.617397 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.617369    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:16.707766 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.707733    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:16.775081 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.775058    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:25:16.798666 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.798588    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:16.815225 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.815203    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67234690-8c0f-4bc7-8e4d-abd82315ce9a-maas-api-tls\") pod \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") "
Jul 29 21:25:16.815317 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.815234    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67234690-8c0f-4bc7-8e4d-abd82315ce9a-openshift-service-ca\") pod \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") "
Jul 29 21:25:16.815317 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.815262    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvwzz\" (UniqueName: \"kubernetes.io/projected/67234690-8c0f-4bc7-8e4d-abd82315ce9a-kube-api-access-dvwzz\") pod \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\" (UID: \"67234690-8c0f-4bc7-8e4d-abd82315ce9a\") "
Jul 29 21:25:16.815628 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.815603    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67234690-8c0f-4bc7-8e4d-abd82315ce9a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67234690-8c0f-4bc7-8e4d-abd82315ce9a" (UID: "67234690-8c0f-4bc7-8e4d-abd82315ce9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:25:16.817308 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.817281    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67234690-8c0f-4bc7-8e4d-abd82315ce9a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67234690-8c0f-4bc7-8e4d-abd82315ce9a" (UID: "67234690-8c0f-4bc7-8e4d-abd82315ce9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:25:16.817425 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.817372    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67234690-8c0f-4bc7-8e4d-abd82315ce9a-kube-api-access-dvwzz" (OuterVolumeSpecName: "kube-api-access-dvwzz") pod "67234690-8c0f-4bc7-8e4d-abd82315ce9a" (UID: "67234690-8c0f-4bc7-8e4d-abd82315ce9a"). InnerVolumeSpecName "kube-api-access-dvwzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:25:16.916038 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.916013    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67234690-8c0f-4bc7-8e4d-abd82315ce9a-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:25:16.916038 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.916036    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67234690-8c0f-4bc7-8e4d-abd82315ce9a-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:25:16.916172 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.916047    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvwzz\" (UniqueName: \"kubernetes.io/projected/67234690-8c0f-4bc7-8e4d-abd82315ce9a-kube-api-access-dvwzz\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:25:16.985678 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.985652    2585 generic.go:356] "Generic (PLEG): container finished" podID="67234690-8c0f-4bc7-8e4d-abd82315ce9a" containerID="6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e" exitCode=0
Jul 29 21:25:16.985827 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.985694    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc" event={"ID":"67234690-8c0f-4bc7-8e4d-abd82315ce9a","Type":"ContainerDied","Data":"6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e"}
Jul 29 21:25:16.985827 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.985702    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"
Jul 29 21:25:16.985827 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.985735    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc" event={"ID":"67234690-8c0f-4bc7-8e4d-abd82315ce9a","Type":"ContainerDied","Data":"ace98c2dc069c412ffa33576b461522cc582c86cbeda100c6bc1cb3733c60614"}
Jul 29 21:25:16.985827 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.985756    2585 scope.go:117] "RemoveContainer" containerID="6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e"
Jul 29 21:25:16.994889 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.994870    2585 scope.go:117] "RemoveContainer" containerID="6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e"
Jul 29 21:25:16.995153 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:25:16.995134    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e\": container with ID starting with 6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e not found: ID does not exist" containerID="6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e"
Jul 29 21:25:16.995198 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:16.995161    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e"} err="failed to get container status \"6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e\": rpc error: code = NotFound desc = could not find container \"6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e\": container with ID starting with 6abbcb1e788fc73e93062a221d37b5a15c0ff1391daa37e1e3e8c0af9b92eb2e not found: ID does not exist"
Jul 29 21:25:17.009870 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:17.009847    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"]
Jul 29 21:25:17.016201 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:17.016178    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aa96fae3-69cd845f56-nbpqc"]
Jul 29 21:25:17.290511 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:17.290450    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:17.523829 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:17.523789    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67234690-8c0f-4bc7-8e4d-abd82315ce9a" path="/var/lib/kubelet/pods/67234690-8c0f-4bc7-8e4d-abd82315ce9a/volumes"
Jul 29 21:25:22.900771 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:22.900735    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:22.998481 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:22.998450    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:23.185760 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:23.185687    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:23.392269 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:23.392241    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:23.562488 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:23.562457    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:23.653047 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:23.653014    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:33.351508 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:33.351467    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:34.163827 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:34.163755    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:34.422988 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:34.422925    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:25:55.391258 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:25:55.391225    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:00.566969 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:00.566875    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:01.738976 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:01.738945    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"]
Jul 29 21:26:01.739739 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:01.739194    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7" podUID="be190b1d-ad17-4a0e-8cfd-aacefbff6551" containerName="maas-api" containerID="cri-o://5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a" gracePeriod=30
Jul 29 21:26:01.766827 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:01.766806    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:01.838660 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:01.838617    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:01.908173 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:01.908142    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:02.007279 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.007215    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:26:02.084046 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.084011    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqxcq\" (UniqueName: \"kubernetes.io/projected/be190b1d-ad17-4a0e-8cfd-aacefbff6551-kube-api-access-wqxcq\") pod \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") "
Jul 29 21:26:02.084239 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.084096    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be190b1d-ad17-4a0e-8cfd-aacefbff6551-openshift-service-ca\") pod \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") "
Jul 29 21:26:02.084239 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.084142    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be190b1d-ad17-4a0e-8cfd-aacefbff6551-maas-api-tls\") pod \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\" (UID: \"be190b1d-ad17-4a0e-8cfd-aacefbff6551\") "
Jul 29 21:26:02.084469 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.084439    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be190b1d-ad17-4a0e-8cfd-aacefbff6551-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be190b1d-ad17-4a0e-8cfd-aacefbff6551" (UID: "be190b1d-ad17-4a0e-8cfd-aacefbff6551"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:26:02.086250 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.086224    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be190b1d-ad17-4a0e-8cfd-aacefbff6551-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be190b1d-ad17-4a0e-8cfd-aacefbff6551" (UID: "be190b1d-ad17-4a0e-8cfd-aacefbff6551"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:26:02.086438 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.086252    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be190b1d-ad17-4a0e-8cfd-aacefbff6551-kube-api-access-wqxcq" (OuterVolumeSpecName: "kube-api-access-wqxcq") pod "be190b1d-ad17-4a0e-8cfd-aacefbff6551" (UID: "be190b1d-ad17-4a0e-8cfd-aacefbff6551"). InnerVolumeSpecName "kube-api-access-wqxcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:26:02.185123 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.185086    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxcq\" (UniqueName: \"kubernetes.io/projected/be190b1d-ad17-4a0e-8cfd-aacefbff6551-kube-api-access-wqxcq\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:26:02.185123 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.185122    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be190b1d-ad17-4a0e-8cfd-aacefbff6551-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:26:02.185317 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.185139    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be190b1d-ad17-4a0e-8cfd-aacefbff6551-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:26:02.204915 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.204885    2585 generic.go:356] "Generic (PLEG): container finished" podID="be190b1d-ad17-4a0e-8cfd-aacefbff6551" containerID="5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a" exitCode=0
Jul 29 21:26:02.205069 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.204919    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7" event={"ID":"be190b1d-ad17-4a0e-8cfd-aacefbff6551","Type":"ContainerDied","Data":"5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a"}
Jul 29 21:26:02.205069 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.204952    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"
Jul 29 21:26:02.205069 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.205018    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7" event={"ID":"be190b1d-ad17-4a0e-8cfd-aacefbff6551","Type":"ContainerDied","Data":"27197c7de7966333e52fa3a37af6d825cfdde0450d2d306e19f79f086b524f70"}
Jul 29 21:26:02.205069 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.205055    2585 scope.go:117] "RemoveContainer" containerID="5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a"
Jul 29 21:26:02.216714 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.216696    2585 scope.go:117] "RemoveContainer" containerID="5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a"
Jul 29 21:26:02.217007 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:26:02.216981    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a\": container with ID starting with 5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a not found: ID does not exist" containerID="5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a"
Jul 29 21:26:02.217140 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.217013    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a"} err="failed to get container status \"5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a\": rpc error: code = NotFound desc = could not find container \"5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a\": container with ID starting with 5c30ea4f64f374840ed9cafc29a074d3b70bd914ed2f075452901de4fe39725a not found: ID does not exist"
Jul 29 21:26:02.230214 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.230188    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"]
Jul 29 21:26:02.234556 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.234533    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcb1aff7-59584cddcd-4rqw7"]
Jul 29 21:26:02.347587 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.347510    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:02.682894 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:02.682821    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:03.435934 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:03.435901    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:03.525703 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:03.525671    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be190b1d-ad17-4a0e-8cfd-aacefbff6551" path="/var/lib/kubelet/pods/be190b1d-ad17-4a0e-8cfd-aacefbff6551/volumes"
Jul 29 21:26:03.536106 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:03.536076    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:05.995908 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:05.995848    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:06.075089 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.075059    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"]
Jul 29 21:26:06.075475 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.075460    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67234690-8c0f-4bc7-8e4d-abd82315ce9a" containerName="maas-api"
Jul 29 21:26:06.075475 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.075475    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="67234690-8c0f-4bc7-8e4d-abd82315ce9a" containerName="maas-api"
Jul 29 21:26:06.075599 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.075490    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be190b1d-ad17-4a0e-8cfd-aacefbff6551" containerName="maas-api"
Jul 29 21:26:06.075599 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.075496    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="be190b1d-ad17-4a0e-8cfd-aacefbff6551" containerName="maas-api"
Jul 29 21:26:06.075599 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.075559    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="be190b1d-ad17-4a0e-8cfd-aacefbff6551" containerName="maas-api"
Jul 29 21:26:06.075599 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.075567    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="67234690-8c0f-4bc7-8e4d-abd82315ce9a" containerName="maas-api"
Jul 29 21:26:06.080026 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.080009    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.082912 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.082896    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b04ec1-serving-cert\""
Jul 29 21:26:06.091436 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.091412    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"]
Jul 29 21:26:06.114171 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.114147    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:06.187925 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.187899    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:06.217055 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.217028    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a1310e-2af4-4297-a212-4ae44955a2c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.217157 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.217070    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a1310e-2af4-4297-a212-4ae44955a2c8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.217157 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.217133    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmz7p\" (UniqueName: \"kubernetes.io/projected/81a1310e-2af4-4297-a212-4ae44955a2c8-kube-api-access-qmz7p\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.318539 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.318479    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a1310e-2af4-4297-a212-4ae44955a2c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.318539 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.318511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a1310e-2af4-4297-a212-4ae44955a2c8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.318666 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:26:06.318629    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-serving-cert: secret "maas-api-e2e-inf-a-b04ec1-serving-cert" not found
Jul 29 21:26:06.318709 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.318658    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmz7p\" (UniqueName: \"kubernetes.io/projected/81a1310e-2af4-4297-a212-4ae44955a2c8-kube-api-access-qmz7p\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.318709 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:26:06.318691    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a1310e-2af4-4297-a212-4ae44955a2c8-maas-api-tls podName:81a1310e-2af4-4297-a212-4ae44955a2c8 nodeName:}" failed. No retries permitted until 2026-07-29 21:26:06.81867142 +0000 UTC m=+3693.862363582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81a1310e-2af4-4297-a212-4ae44955a2c8-maas-api-tls") pod "maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6" (UID: "81a1310e-2af4-4297-a212-4ae44955a2c8") : secret "maas-api-e2e-inf-a-b04ec1-serving-cert" not found
Jul 29 21:26:06.319109 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.319092    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a1310e-2af4-4297-a212-4ae44955a2c8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.326697 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.326679    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmz7p\" (UniqueName: \"kubernetes.io/projected/81a1310e-2af4-4297-a212-4ae44955a2c8-kube-api-access-qmz7p\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.378195 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.378175    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:06.684098 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.684012    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:06.785242 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.785206    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:06.822575 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.822535    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a1310e-2af4-4297-a212-4ae44955a2c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.824954 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.824925    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a1310e-2af4-4297-a212-4ae44955a2c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:06.991856 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:06.991829    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:07.337901 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:07.337877    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"]
Jul 29 21:26:07.340092 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:26:07.340063    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a1310e_2af4_4297_a212_4ae44955a2c8.slice/crio-17e0b2be3a0f8f45b71ddb721b12c717a6d46996711b046c6679eba2e70873b9 WatchSource:0}: Error finding container 17e0b2be3a0f8f45b71ddb721b12c717a6d46996711b046c6679eba2e70873b9: Status 404 returned error can't find the container with id 17e0b2be3a0f8f45b71ddb721b12c717a6d46996711b046c6679eba2e70873b9
Jul 29 21:26:07.341830 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:07.341813    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:26:08.241623 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:08.241588    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6" event={"ID":"81a1310e-2af4-4297-a212-4ae44955a2c8","Type":"ContainerStarted","Data":"83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880"}
Jul 29 21:26:08.241762 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:08.241664    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6" event={"ID":"81a1310e-2af4-4297-a212-4ae44955a2c8","Type":"ContainerStarted","Data":"17e0b2be3a0f8f45b71ddb721b12c717a6d46996711b046c6679eba2e70873b9"}
Jul 29 21:26:08.241762 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:08.241705    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:08.279096 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:08.279046    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6" podStartSLOduration=1.589352629 podStartE2EDuration="2.279032501s" totalImagesPullingTime="689.679872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:08.276731432 +0000 UTC m=+3695.320423616" watchObservedRunningTime="2026-07-29 21:26:08.279032501 +0000 UTC m=+3695.322724684"
Jul 29 21:26:08.675554 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:08.675521    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:08.775434 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:08.775403    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:14.251061 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:14.251033    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:26:31.029000 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:31.028962    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:31.381350 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:31.381252    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:32.071850 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:32.071818    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:32.139862 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:32.139833    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:35.159506 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.159459    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:35.240599 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.240565    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:35.245106 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.245078    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"]
Jul 29 21:26:35.249871 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.249847    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.253693 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.253670    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-46f813-serving-cert\""
Jul 29 21:26:35.260589 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.260565    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"]
Jul 29 21:26:35.308379 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.308355    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:35.351836 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.351810    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a7103-18ba-4c81-9e01-835410c8fb9b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.351926 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.351874    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a7103-18ba-4c81-9e01-835410c8fb9b-maas-api-tls\") pod \"maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.351972 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.351924    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7x2h\" (UniqueName: \"kubernetes.io/projected/6c9a7103-18ba-4c81-9e01-835410c8fb9b-kube-api-access-k7x2h\") pod \"maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.408223 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.408199    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:35.452525 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.452471    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a7103-18ba-4c81-9e01-835410c8fb9b-maas-api-tls\") pod \"maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.452525 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.452516    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7x2h\" (UniqueName: \"kubernetes.io/projected/6c9a7103-18ba-4c81-9e01-835410c8fb9b-kube-api-access-k7x2h\") pod \"maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.452638 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.452554    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a7103-18ba-4c81-9e01-835410c8fb9b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.453113 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.453095    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a7103-18ba-4c81-9e01-835410c8fb9b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.454967 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.454943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a7103-18ba-4c81-9e01-835410c8fb9b-maas-api-tls\") pod \"maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.460890 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.460870    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7x2h\" (UniqueName: \"kubernetes.io/projected/6c9a7103-18ba-4c81-9e01-835410c8fb9b-kube-api-access-k7x2h\") pod \"maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.562552 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.562528    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:35.694628 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.694600    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"]
Jul 29 21:26:35.696305 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:26:35.696263    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9a7103_18ba_4c81_9e01_835410c8fb9b.slice/crio-d492de3f5ed4016e57dd388fe23fd3cf00d66b895a5ec565985ed905f8651491 WatchSource:0}: Error finding container d492de3f5ed4016e57dd388fe23fd3cf00d66b895a5ec565985ed905f8651491: Status 404 returned error can't find the container with id d492de3f5ed4016e57dd388fe23fd3cf00d66b895a5ec565985ed905f8651491
Jul 29 21:26:35.878059 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.878027    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:35.971784 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:35.971752    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:26:36.377431 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:36.377396    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4" event={"ID":"6c9a7103-18ba-4c81-9e01-835410c8fb9b","Type":"ContainerStarted","Data":"1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834"}
Jul 29 21:26:36.377816 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:36.377453    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4" event={"ID":"6c9a7103-18ba-4c81-9e01-835410c8fb9b","Type":"ContainerStarted","Data":"d492de3f5ed4016e57dd388fe23fd3cf00d66b895a5ec565985ed905f8651491"}
Jul 29 21:26:36.377816 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:36.377584    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:26:36.396982 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:36.396938    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4" podStartSLOduration=0.935661158 podStartE2EDuration="1.396923399s" totalImagesPullingTime="461.262241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:36.395459925 +0000 UTC m=+3723.439152101" watchObservedRunningTime="2026-07-29 21:26:36.396923399 +0000 UTC m=+3723.440615582"
Jul 29 21:26:42.388370 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:26:42.388318    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:27:00.161193 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:00.161163    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:27:05.449800 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:05.449759    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:27:46.539191 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.539111    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"]
Jul 29 21:27:46.542920 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.542901    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.547692 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.547666    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-46f813\"/\"kube-root-ca.crt\""
Jul 29 21:27:46.547811 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.547665    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-46f813\"/\"default-dockercfg-qvst9\""
Jul 29 21:27:46.547811 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.547717    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-46f813\"/\"test-model-46f813-kserve-self-signed-certs\""
Jul 29 21:27:46.547811 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.547752    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-46f813\"/\"openshift-service-ca.crt\""
Jul 29 21:27:46.554514 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.554494    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"]
Jul 29 21:27:46.655635 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.655601    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1eaf4c-7348-40e2-8146-2c48146ae379-tls-certs\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.655804 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.655691    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-tmp-dir\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.655804 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.655715    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-dshm\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.655804 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.655749    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-model-cache\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.655804 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.655784    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wpb5\" (UniqueName: \"kubernetes.io/projected/8e1eaf4c-7348-40e2-8146-2c48146ae379-kube-api-access-5wpb5\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.655947 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.655808    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-home\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.736069 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.736038    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:27:46.757032 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.757006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-tmp-dir\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.757128 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.757037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-dshm\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.757128 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.757076    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-model-cache\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.757207 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.757188    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wpb5\" (UniqueName: \"kubernetes.io/projected/8e1eaf4c-7348-40e2-8146-2c48146ae379-kube-api-access-5wpb5\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.757254 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.757220    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-home\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.757312 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.757276    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1eaf4c-7348-40e2-8146-2c48146ae379-tls-certs\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.757473 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.757454    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-model-cache\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.757559 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.757487    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-tmp-dir\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.757647 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.757626    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-home\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.759247 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.759229    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-dshm\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.759638 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.759622    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1eaf4c-7348-40e2-8146-2c48146ae379-tls-certs\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.768531 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.768507    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wpb5\" (UniqueName: \"kubernetes.io/projected/8e1eaf4c-7348-40e2-8146-2c48146ae379-kube-api-access-5wpb5\") pod \"test-model-46f813-kserve-64c6785549-pcmsf\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") " pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.852372 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.852290    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:46.988961 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:46.988824    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"]
Jul 29 21:27:46.991140 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:27:46.991117    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e1eaf4c_7348_40e2_8146_2c48146ae379.slice/crio-2963e25860b130d01f8d54079ea7e2081c324f2633aa3ef3eab68f895e4bec8d WatchSource:0}: Error finding container 2963e25860b130d01f8d54079ea7e2081c324f2633aa3ef3eab68f895e4bec8d: Status 404 returned error can't find the container with id 2963e25860b130d01f8d54079ea7e2081c324f2633aa3ef3eab68f895e4bec8d
Jul 29 21:27:47.709445 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:47.709411    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf" event={"ID":"8e1eaf4c-7348-40e2-8146-2c48146ae379","Type":"ContainerStarted","Data":"526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387"}
Jul 29 21:27:47.709445 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:47.709451    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf" event={"ID":"8e1eaf4c-7348-40e2-8146-2c48146ae379","Type":"ContainerStarted","Data":"2963e25860b130d01f8d54079ea7e2081c324f2633aa3ef3eab68f895e4bec8d"}
Jul 29 21:27:47.734691 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:47.734647    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf" podStartSLOduration=1.734631717 podStartE2EDuration="1.734631717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:27:47.733113253 +0000 UTC m=+3794.776805437" watchObservedRunningTime="2026-07-29 21:27:47.734631717 +0000 UTC m=+3794.778323900"
Jul 29 21:27:56.852988 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:56.852951    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:56.853456 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:56.853000    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:56.865567 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:56.865543    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:27:57.768116 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:27:57.768087    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:28:35.737172 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:35.737135    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:36.277143 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:36.277113    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"]
Jul 29 21:28:36.349124 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:36.349096    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:47.017431 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.017396    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"]
Jul 29 21:28:47.017994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.017681    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6" podUID="81a1310e-2af4-4297-a212-4ae44955a2c8" containerName="maas-api" containerID="cri-o://83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880" gracePeriod=30
Jul 29 21:28:47.083661 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.083638    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:47.270553 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.270495    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:28:47.270874 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.270854    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:47.355722 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.355693    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a1310e-2af4-4297-a212-4ae44955a2c8-openshift-service-ca\") pod \"81a1310e-2af4-4297-a212-4ae44955a2c8\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") "
Jul 29 21:28:47.355856 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.355746    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a1310e-2af4-4297-a212-4ae44955a2c8-maas-api-tls\") pod \"81a1310e-2af4-4297-a212-4ae44955a2c8\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") "
Jul 29 21:28:47.355856 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.355778    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmz7p\" (UniqueName: \"kubernetes.io/projected/81a1310e-2af4-4297-a212-4ae44955a2c8-kube-api-access-qmz7p\") pod \"81a1310e-2af4-4297-a212-4ae44955a2c8\" (UID: \"81a1310e-2af4-4297-a212-4ae44955a2c8\") "
Jul 29 21:28:47.356118 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.356093    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a1310e-2af4-4297-a212-4ae44955a2c8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81a1310e-2af4-4297-a212-4ae44955a2c8" (UID: "81a1310e-2af4-4297-a212-4ae44955a2c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:28:47.357934 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.357910    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a1310e-2af4-4297-a212-4ae44955a2c8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81a1310e-2af4-4297-a212-4ae44955a2c8" (UID: "81a1310e-2af4-4297-a212-4ae44955a2c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:28:47.358039 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.357980    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a1310e-2af4-4297-a212-4ae44955a2c8-kube-api-access-qmz7p" (OuterVolumeSpecName: "kube-api-access-qmz7p") pod "81a1310e-2af4-4297-a212-4ae44955a2c8" (UID: "81a1310e-2af4-4297-a212-4ae44955a2c8"). InnerVolumeSpecName "kube-api-access-qmz7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:28:47.457432 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.457405    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a1310e-2af4-4297-a212-4ae44955a2c8-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:28:47.457432 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.457429    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81a1310e-2af4-4297-a212-4ae44955a2c8-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:28:47.457580 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.457439    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmz7p\" (UniqueName: \"kubernetes.io/projected/81a1310e-2af4-4297-a212-4ae44955a2c8-kube-api-access-qmz7p\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:28:47.537462 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.537434    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:47.671638 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:47.671608    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:48.003424 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.003392    2585 generic.go:356] "Generic (PLEG): container finished" podID="81a1310e-2af4-4297-a212-4ae44955a2c8" containerID="83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880" exitCode=0
Jul 29 21:28:48.003576 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.003429    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6" event={"ID":"81a1310e-2af4-4297-a212-4ae44955a2c8","Type":"ContainerDied","Data":"83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880"}
Jul 29 21:28:48.003576 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.003451    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"
Jul 29 21:28:48.003576 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.003466    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6" event={"ID":"81a1310e-2af4-4297-a212-4ae44955a2c8","Type":"ContainerDied","Data":"17e0b2be3a0f8f45b71ddb721b12c717a6d46996711b046c6679eba2e70873b9"}
Jul 29 21:28:48.003576 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.003488    2585 scope.go:117] "RemoveContainer" containerID="83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880"
Jul 29 21:28:48.013487 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.013470    2585 scope.go:117] "RemoveContainer" containerID="83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880"
Jul 29 21:28:48.013768 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:28:48.013745    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880\": container with ID starting with 83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880 not found: ID does not exist" containerID="83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880"
Jul 29 21:28:48.013845 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.013780    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880"} err="failed to get container status \"83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880\": rpc error: code = NotFound desc = could not find container \"83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880\": container with ID starting with 83e9cbd85199c6eb7ec6f73254f6ae0456c644e4cf135aff5bae5d0ad5621880 not found: ID does not exist"
Jul 29 21:28:48.034136 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.034115    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:48.040390 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.040371    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"]
Jul 29 21:28:48.049488 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:48.049470    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b04ec1-5cdb7f8bfc-rvqg6"]
Jul 29 21:28:49.525033 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:49.524993    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81a1310e-2af4-4297-a212-4ae44955a2c8" path="/var/lib/kubelet/pods/81a1310e-2af4-4297-a212-4ae44955a2c8/volumes"
Jul 29 21:28:51.296003 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.295967    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf" podUID="8e1eaf4c-7348-40e2-8146-2c48146ae379" containerName="main" containerID="cri-o://526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387" gracePeriod=45
Jul 29 21:28:51.544175 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.544149    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:28:51.697107 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.697044    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1eaf4c-7348-40e2-8146-2c48146ae379-tls-certs\") pod \"8e1eaf4c-7348-40e2-8146-2c48146ae379\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") "
Jul 29 21:28:51.697107 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.697078    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-home\") pod \"8e1eaf4c-7348-40e2-8146-2c48146ae379\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") "
Jul 29 21:28:51.697306 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.697117    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-model-cache\") pod \"8e1eaf4c-7348-40e2-8146-2c48146ae379\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") "
Jul 29 21:28:51.697306 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.697141    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-tmp-dir\") pod \"8e1eaf4c-7348-40e2-8146-2c48146ae379\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") "
Jul 29 21:28:51.697306 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.697188    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-dshm\") pod \"8e1eaf4c-7348-40e2-8146-2c48146ae379\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") "
Jul 29 21:28:51.697306 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.697204    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wpb5\" (UniqueName: \"kubernetes.io/projected/8e1eaf4c-7348-40e2-8146-2c48146ae379-kube-api-access-5wpb5\") pod \"8e1eaf4c-7348-40e2-8146-2c48146ae379\" (UID: \"8e1eaf4c-7348-40e2-8146-2c48146ae379\") "
Jul 29 21:28:51.697547 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.697432    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-home" (OuterVolumeSpecName: "home") pod "8e1eaf4c-7348-40e2-8146-2c48146ae379" (UID: "8e1eaf4c-7348-40e2-8146-2c48146ae379"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:28:51.697547 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.697481    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e1eaf4c-7348-40e2-8146-2c48146ae379" (UID: "8e1eaf4c-7348-40e2-8146-2c48146ae379"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:28:51.697547 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.697498    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e1eaf4c-7348-40e2-8146-2c48146ae379" (UID: "8e1eaf4c-7348-40e2-8146-2c48146ae379"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:28:51.699259 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.699228    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-dshm" (OuterVolumeSpecName: "dshm") pod "8e1eaf4c-7348-40e2-8146-2c48146ae379" (UID: "8e1eaf4c-7348-40e2-8146-2c48146ae379"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:28:51.699378 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.699295    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1eaf4c-7348-40e2-8146-2c48146ae379-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e1eaf4c-7348-40e2-8146-2c48146ae379" (UID: "8e1eaf4c-7348-40e2-8146-2c48146ae379"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:28:51.699378 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.699302    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1eaf4c-7348-40e2-8146-2c48146ae379-kube-api-access-5wpb5" (OuterVolumeSpecName: "kube-api-access-5wpb5") pod "8e1eaf4c-7348-40e2-8146-2c48146ae379" (UID: "8e1eaf4c-7348-40e2-8146-2c48146ae379"). InnerVolumeSpecName "kube-api-access-5wpb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:28:51.797911 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.797882    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1eaf4c-7348-40e2-8146-2c48146ae379-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:28:51.797911 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.797906    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:28:51.797911 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.797914    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:28:51.798106 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.797941    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:28:51.798106 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.797948    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1eaf4c-7348-40e2-8146-2c48146ae379-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:28:51.798106 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:51.797957    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wpb5\" (UniqueName: \"kubernetes.io/projected/8e1eaf4c-7348-40e2-8146-2c48146ae379-kube-api-access-5wpb5\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:28:52.023578 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:52.023546    2585 generic.go:356] "Generic (PLEG): container finished" podID="8e1eaf4c-7348-40e2-8146-2c48146ae379" containerID="526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387" exitCode=0
Jul 29 21:28:52.023729 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:52.023600    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"
Jul 29 21:28:52.023729 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:52.023629    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf" event={"ID":"8e1eaf4c-7348-40e2-8146-2c48146ae379","Type":"ContainerDied","Data":"526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387"}
Jul 29 21:28:52.023729 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:52.023672    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf" event={"ID":"8e1eaf4c-7348-40e2-8146-2c48146ae379","Type":"ContainerDied","Data":"2963e25860b130d01f8d54079ea7e2081c324f2633aa3ef3eab68f895e4bec8d"}
Jul 29 21:28:52.023729 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:52.023690    2585 scope.go:117] "RemoveContainer" containerID="526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387"
Jul 29 21:28:52.033226 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:52.033210    2585 scope.go:117] "RemoveContainer" containerID="526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387"
Jul 29 21:28:52.033532 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:28:52.033503    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387\": container with ID starting with 526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387 not found: ID does not exist" containerID="526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387"
Jul 29 21:28:52.033636 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:52.033538    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387"} err="failed to get container status \"526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387\": rpc error: code = NotFound desc = could not find container \"526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387\": container with ID starting with 526e15ddb943422d3269a42c19d812beac3faa3daa8d947db053871afa228387 not found: ID does not exist"
Jul 29 21:28:52.047158 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:52.047134    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"]
Jul 29 21:28:52.051910 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:52.051887    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-46f813/test-model-46f813-kserve-64c6785549-pcmsf"]
Jul 29 21:28:53.524076 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:53.524034    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1eaf4c-7348-40e2-8146-2c48146ae379" path="/var/lib/kubelet/pods/8e1eaf4c-7348-40e2-8146-2c48146ae379/volumes"
Jul 29 21:28:54.399784 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:54.399752    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:54.487306 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:54.487270    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:54.697409 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:54.697315    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:55.037201 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.037167    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"]
Jul 29 21:28:55.037582 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.037568    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81a1310e-2af4-4297-a212-4ae44955a2c8" containerName="maas-api"
Jul 29 21:28:55.037647 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.037584    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="81a1310e-2af4-4297-a212-4ae44955a2c8" containerName="maas-api"
Jul 29 21:28:55.037647 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.037595    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1eaf4c-7348-40e2-8146-2c48146ae379" containerName="main"
Jul 29 21:28:55.037647 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.037600    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1eaf4c-7348-40e2-8146-2c48146ae379" containerName="main"
Jul 29 21:28:55.037758 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.037663    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="81a1310e-2af4-4297-a212-4ae44955a2c8" containerName="maas-api"
Jul 29 21:28:55.037758 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.037677    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1eaf4c-7348-40e2-8146-2c48146ae379" containerName="main"
Jul 29 21:28:55.041876 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.041857    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"
Jul 29 21:28:55.045480 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.045461    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wpqrc\""
Jul 29 21:28:55.049531 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.049504    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"]
Jul 29 21:28:55.118883 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.118854    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:28:55.124284 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.124263    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823a84ca-0af3-461f-b569-7f65d6f0dda4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-46f813-26n7l\" (UID: \"823a84ca-0af3-461f-b569-7f65d6f0dda4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"
Jul 29 21:28:55.225723 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.225688    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823a84ca-0af3-461f-b569-7f65d6f0dda4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-46f813-26n7l\" (UID: \"823a84ca-0af3-461f-b569-7f65d6f0dda4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"
Jul 29 21:28:55.226308 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.226287    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823a84ca-0af3-461f-b569-7f65d6f0dda4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-46f813-26n7l\" (UID: \"823a84ca-0af3-461f-b569-7f65d6f0dda4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"
Jul 29 21:28:55.353475 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.353397    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"
Jul 29 21:28:55.517440 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:55.517411    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"]
Jul 29 21:28:55.518987 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:28:55.518960    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823a84ca_0af3_461f_b569_7f65d6f0dda4.slice/crio-e1fdb9d1d2e915f7dd32daded34e3efd2b773526bb1a76c2db5e6afc1bc8beb4 WatchSource:0}: Error finding container e1fdb9d1d2e915f7dd32daded34e3efd2b773526bb1a76c2db5e6afc1bc8beb4: Status 404 returned error can't find the container with id e1fdb9d1d2e915f7dd32daded34e3efd2b773526bb1a76c2db5e6afc1bc8beb4
Jul 29 21:28:56.045966 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:56.045931    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l" event={"ID":"823a84ca-0af3-461f-b569-7f65d6f0dda4","Type":"ContainerStarted","Data":"e1fdb9d1d2e915f7dd32daded34e3efd2b773526bb1a76c2db5e6afc1bc8beb4"}
Jul 29 21:28:58.057409 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:58.057382    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l" event={"ID":"823a84ca-0af3-461f-b569-7f65d6f0dda4","Type":"ContainerStarted","Data":"df34fa9e5e6789d08d23713d400b5a7de9c31e355341bb24ffd41e3aaf8ce6cb"}
Jul 29 21:28:58.080302 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:58.080255    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l" podStartSLOduration=0.961173866 podStartE2EDuration="3.080242869s" totalImagesPullingTime="2.119069003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:28:58.07872369 +0000 UTC m=+3865.122415873" watchObservedRunningTime="2026-07-29 21:28:58.080242869 +0000 UTC m=+3865.123935084"
Jul 29 21:28:59.063904 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:59.063869    2585 generic.go:356] "Generic (PLEG): container finished" podID="823a84ca-0af3-461f-b569-7f65d6f0dda4" containerID="df34fa9e5e6789d08d23713d400b5a7de9c31e355341bb24ffd41e3aaf8ce6cb" exitCode=0
Jul 29 21:28:59.064271 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:28:59.063963    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l" event={"ID":"823a84ca-0af3-461f-b569-7f65d6f0dda4","Type":"ContainerDied","Data":"df34fa9e5e6789d08d23713d400b5a7de9c31e355341bb24ffd41e3aaf8ce6cb"}
Jul 29 21:29:00.199136 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:00.199114    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"
Jul 29 21:29:00.366552 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:00.366482    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823a84ca-0af3-461f-b569-7f65d6f0dda4-maas-api-service-ca\") pod \"823a84ca-0af3-461f-b569-7f65d6f0dda4\" (UID: \"823a84ca-0af3-461f-b569-7f65d6f0dda4\") "
Jul 29 21:29:00.366792 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:00.366770    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823a84ca-0af3-461f-b569-7f65d6f0dda4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "823a84ca-0af3-461f-b569-7f65d6f0dda4" (UID: "823a84ca-0af3-461f-b569-7f65d6f0dda4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:29:00.467989 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:00.467958    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823a84ca-0af3-461f-b569-7f65d6f0dda4-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:29:01.075244 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:01.075209    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"
Jul 29 21:29:01.075403 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:01.075244    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l" event={"ID":"823a84ca-0af3-461f-b569-7f65d6f0dda4","Type":"ContainerDied","Data":"e1fdb9d1d2e915f7dd32daded34e3efd2b773526bb1a76c2db5e6afc1bc8beb4"}
Jul 29 21:29:01.075403 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:01.075300    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1fdb9d1d2e915f7dd32daded34e3efd2b773526bb1a76c2db5e6afc1bc8beb4"
Jul 29 21:29:05.527679 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.527648    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"]
Jul 29 21:29:05.528127 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.527823    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4" podUID="6c9a7103-18ba-4c81-9e01-835410c8fb9b" containerName="maas-api" containerID="cri-o://1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834" gracePeriod=30
Jul 29 21:29:05.557142 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.557117    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:05.629122 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.629097    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:05.723803 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.723775    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:05.738810 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.738760    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"]
Jul 29 21:29:05.740969 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.740945    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46f813-26n7l"]
Jul 29 21:29:05.774356 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.774318    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:29:05.783164 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.783122    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:05.912287 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.912238    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a7103-18ba-4c81-9e01-835410c8fb9b-maas-api-tls\") pod \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") "
Jul 29 21:29:05.912482 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.912458    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7x2h\" (UniqueName: \"kubernetes.io/projected/6c9a7103-18ba-4c81-9e01-835410c8fb9b-kube-api-access-k7x2h\") pod \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") "
Jul 29 21:29:05.912615 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.912591    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a7103-18ba-4c81-9e01-835410c8fb9b-openshift-service-ca\") pod \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\" (UID: \"6c9a7103-18ba-4c81-9e01-835410c8fb9b\") "
Jul 29 21:29:05.912985 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.912950    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9a7103-18ba-4c81-9e01-835410c8fb9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c9a7103-18ba-4c81-9e01-835410c8fb9b" (UID: "6c9a7103-18ba-4c81-9e01-835410c8fb9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:29:05.914664 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.914628    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9a7103-18ba-4c81-9e01-835410c8fb9b-kube-api-access-k7x2h" (OuterVolumeSpecName: "kube-api-access-k7x2h") pod "6c9a7103-18ba-4c81-9e01-835410c8fb9b" (UID: "6c9a7103-18ba-4c81-9e01-835410c8fb9b"). InnerVolumeSpecName "kube-api-access-k7x2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:29:05.914664 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:05.914633    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9a7103-18ba-4c81-9e01-835410c8fb9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c9a7103-18ba-4c81-9e01-835410c8fb9b" (UID: "6c9a7103-18ba-4c81-9e01-835410c8fb9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:29:06.013813 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.013785    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a7103-18ba-4c81-9e01-835410c8fb9b-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:29:06.013813 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.013811    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a7103-18ba-4c81-9e01-835410c8fb9b-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:29:06.013939 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.013824    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7x2h\" (UniqueName: \"kubernetes.io/projected/6c9a7103-18ba-4c81-9e01-835410c8fb9b-kube-api-access-k7x2h\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:29:06.100234 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.100175    2585 generic.go:356] "Generic (PLEG): container finished" podID="6c9a7103-18ba-4c81-9e01-835410c8fb9b" containerID="1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834" exitCode=0
Jul 29 21:29:06.100234 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.100224    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"
Jul 29 21:29:06.100395 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.100238    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4" event={"ID":"6c9a7103-18ba-4c81-9e01-835410c8fb9b","Type":"ContainerDied","Data":"1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834"}
Jul 29 21:29:06.100395 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.100279    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4" event={"ID":"6c9a7103-18ba-4c81-9e01-835410c8fb9b","Type":"ContainerDied","Data":"d492de3f5ed4016e57dd388fe23fd3cf00d66b895a5ec565985ed905f8651491"}
Jul 29 21:29:06.100395 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.100301    2585 scope.go:117] "RemoveContainer" containerID="1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834"
Jul 29 21:29:06.109625 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.109601    2585 scope.go:117] "RemoveContainer" containerID="1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834"
Jul 29 21:29:06.109877 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:29:06.109855    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834\": container with ID starting with 1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834 not found: ID does not exist" containerID="1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834"
Jul 29 21:29:06.109931 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.109885    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834"} err="failed to get container status \"1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834\": rpc error: code = NotFound desc = could not find container \"1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834\": container with ID starting with 1e7dbd8ed248e95c6a1a7af4e588950104b983e493efcf48ff1ca7e622553834 not found: ID does not exist"
Jul 29 21:29:06.123290 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.123264    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"]
Jul 29 21:29:06.127050 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.127032    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46f813-64c7bf6bbc-r6pq4"]
Jul 29 21:29:06.131645 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:06.131626    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:07.524772 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:07.524728    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c9a7103-18ba-4c81-9e01-835410c8fb9b" path="/var/lib/kubelet/pods/6c9a7103-18ba-4c81-9e01-835410c8fb9b/volumes"
Jul 29 21:29:07.525297 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:07.525208    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823a84ca-0af3-461f-b569-7f65d6f0dda4" path="/var/lib/kubelet/pods/823a84ca-0af3-461f-b569-7f65d6f0dda4/volumes"
Jul 29 21:29:09.408192 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:09.408162    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:09.480439 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:09.480407    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:12.169368 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:12.169315    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:12.230715 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:12.230689    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:12.443565 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:12.443487    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:12.658269 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:12.658239    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:13.026800 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:13.026776    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:13.334063 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:13.333981    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:13.560229 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:13.560201    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:16.023747 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.023718    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:16.105322 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.105284    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"]
Jul 29 21:29:16.105809 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.105784    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823a84ca-0af3-461f-b569-7f65d6f0dda4" containerName="revoke-keys"
Jul 29 21:29:16.105809 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.105809    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="823a84ca-0af3-461f-b569-7f65d6f0dda4" containerName="revoke-keys"
Jul 29 21:29:16.105994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.105848    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c9a7103-18ba-4c81-9e01-835410c8fb9b" containerName="maas-api"
Jul 29 21:29:16.105994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.105856    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c9a7103-18ba-4c81-9e01-835410c8fb9b" containerName="maas-api"
Jul 29 21:29:16.105994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.105949    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c9a7103-18ba-4c81-9e01-835410c8fb9b" containerName="maas-api"
Jul 29 21:29:16.105994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.105962    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="823a84ca-0af3-461f-b569-7f65d6f0dda4" containerName="revoke-keys"
Jul 29 21:29:16.110369 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.110347    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.113099 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.113078    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0109d4-serving-cert\""
Jul 29 21:29:16.120779 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.120759    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"]
Jul 29 21:29:16.194857 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.194829    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tgn\" (UniqueName: \"kubernetes.io/projected/17a5ef05-e27a-4f42-9788-6b4734b923b0-kube-api-access-h5tgn\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.194965 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.194875    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a5ef05-e27a-4f42-9788-6b4734b923b0-maas-api-tls\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.194965 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.194909    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ef05-e27a-4f42-9788-6b4734b923b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.203272 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.203254    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:16.295677 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.295628    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tgn\" (UniqueName: \"kubernetes.io/projected/17a5ef05-e27a-4f42-9788-6b4734b923b0-kube-api-access-h5tgn\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.295677 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.295663    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a5ef05-e27a-4f42-9788-6b4734b923b0-maas-api-tls\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.295814 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.295685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ef05-e27a-4f42-9788-6b4734b923b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.295814 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:29:16.295793    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-serving-cert: secret "maas-api-e2e-api-a-0109d4-serving-cert" not found
Jul 29 21:29:16.295883 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:29:16.295865    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a5ef05-e27a-4f42-9788-6b4734b923b0-maas-api-tls podName:17a5ef05-e27a-4f42-9788-6b4734b923b0 nodeName:}" failed. No retries permitted until 2026-07-29 21:29:16.795843584 +0000 UTC m=+3883.839535764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17a5ef05-e27a-4f42-9788-6b4734b923b0-maas-api-tls") pod "maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn" (UID: "17a5ef05-e27a-4f42-9788-6b4734b923b0") : secret "maas-api-e2e-api-a-0109d4-serving-cert" not found
Jul 29 21:29:16.296400 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.296383    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ef05-e27a-4f42-9788-6b4734b923b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.309772 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.309752    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tgn\" (UniqueName: \"kubernetes.io/projected/17a5ef05-e27a-4f42-9788-6b4734b923b0-kube-api-access-h5tgn\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.310519 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.310497    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:16.404954 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.404930    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:16.721055 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.721025    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:16.799375 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.799309    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a5ef05-e27a-4f42-9788-6b4734b923b0-maas-api-tls\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.801683 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.801658    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a5ef05-e27a-4f42-9788-6b4734b923b0-maas-api-tls\") pod \"maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:16.817465 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:16.817059    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:17.022654 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:17.022574    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:17.367436 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:17.367391    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"]
Jul 29 21:29:18.160348 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:18.160252    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn" event={"ID":"17a5ef05-e27a-4f42-9788-6b4734b923b0","Type":"ContainerStarted","Data":"b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579"}
Jul 29 21:29:18.160348 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:18.160306    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn" event={"ID":"17a5ef05-e27a-4f42-9788-6b4734b923b0","Type":"ContainerStarted","Data":"6db28c3db4c1d7bf21eba003f51dd82d35d3f9a423ae94b8a2c9516e72d6f3d1"}
Jul 29 21:29:18.160593 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:18.160354    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:18.184754 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:18.184695    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn" podStartSLOduration=1.6817781790000002 podStartE2EDuration="2.18467608s" totalImagesPullingTime="502.897901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:29:18.182301358 +0000 UTC m=+3885.225993548" watchObservedRunningTime="2026-07-29 21:29:18.18467608 +0000 UTC m=+3885.228368263"
Jul 29 21:29:24.172123 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:24.172093    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:29:33.775039 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:33.775008    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:29:33.779361 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:33.779321    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:29:41.017943 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:41.017904    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:41.499180 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:41.499150    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:41.827091 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:41.827020    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:42.291159 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:42.291128    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:45.321441 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.321404    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:45.358522 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.358485    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"]
Jul 29 21:29:45.368826 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.368346    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:45.371480 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.371454    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dfd5ca-serving-cert\""
Jul 29 21:29:45.373113 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.373086    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"]
Jul 29 21:29:45.400123 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.400095    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:45.435532 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.435499    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65415b0e-eb68-4178-be35-8d3685ccb03e-openshift-service-ca\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:45.435687 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.435576    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwcc\" (UniqueName: \"kubernetes.io/projected/65415b0e-eb68-4178-be35-8d3685ccb03e-kube-api-access-2hwcc\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:45.435687 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.435628    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65415b0e-eb68-4178-be35-8d3685ccb03e-maas-api-tls\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:45.537171 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.537138    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65415b0e-eb68-4178-be35-8d3685ccb03e-openshift-service-ca\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:45.537376 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.537214    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwcc\" (UniqueName: \"kubernetes.io/projected/65415b0e-eb68-4178-be35-8d3685ccb03e-kube-api-access-2hwcc\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:45.537376 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.537276    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65415b0e-eb68-4178-be35-8d3685ccb03e-maas-api-tls\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:45.537504 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:29:45.537392    2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-serving-cert: secret "maas-api-e2e-api-b-dfd5ca-serving-cert" not found
Jul 29 21:29:45.537504 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:29:45.537481    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65415b0e-eb68-4178-be35-8d3685ccb03e-maas-api-tls podName:65415b0e-eb68-4178-be35-8d3685ccb03e nodeName:}" failed. No retries permitted until 2026-07-29 21:29:46.037449235 +0000 UTC m=+3913.081141413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65415b0e-eb68-4178-be35-8d3685ccb03e-maas-api-tls") pod "maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk" (UID: "65415b0e-eb68-4178-be35-8d3685ccb03e") : secret "maas-api-e2e-api-b-dfd5ca-serving-cert" not found
Jul 29 21:29:45.537863 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.537845    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65415b0e-eb68-4178-be35-8d3685ccb03e-openshift-service-ca\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:45.560802 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.560767    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwcc\" (UniqueName: \"kubernetes.io/projected/65415b0e-eb68-4178-be35-8d3685ccb03e-kube-api-access-2hwcc\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:45.689477 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:45.689391    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:46.043408 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:46.043374    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65415b0e-eb68-4178-be35-8d3685ccb03e-maas-api-tls\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:46.045884 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:46.045859    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65415b0e-eb68-4178-be35-8d3685ccb03e-maas-api-tls\") pod \"maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:46.145840 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:46.145813    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:46.247723 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:46.247692    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:29:46.283870 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:46.283842    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:46.424296 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:46.424272    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"]
Jul 29 21:29:46.426548 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:29:46.426519    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65415b0e_eb68_4178_be35_8d3685ccb03e.slice/crio-9b6c1112319125fd8c9761af59215ad06ebd606bf8f3699c1c54f34fcb043518 WatchSource:0}: Error finding container 9b6c1112319125fd8c9761af59215ad06ebd606bf8f3699c1c54f34fcb043518: Status 404 returned error can't find the container with id 9b6c1112319125fd8c9761af59215ad06ebd606bf8f3699c1c54f34fcb043518
Jul 29 21:29:47.309185 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:47.309150    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk" event={"ID":"65415b0e-eb68-4178-be35-8d3685ccb03e","Type":"ContainerStarted","Data":"e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012"}
Jul 29 21:29:47.309377 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:47.309232    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk" event={"ID":"65415b0e-eb68-4178-be35-8d3685ccb03e","Type":"ContainerStarted","Data":"9b6c1112319125fd8c9761af59215ad06ebd606bf8f3699c1c54f34fcb043518"}
Jul 29 21:29:47.309377 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:47.309307    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:29:47.329663 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:47.329615    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk" podStartSLOduration=1.870356606 podStartE2EDuration="2.329600063s" totalImagesPullingTime="459.243457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:29:47.32776138 +0000 UTC m=+3914.371453563" watchObservedRunningTime="2026-07-29 21:29:47.329600063 +0000 UTC m=+3914.373292245"
Jul 29 21:29:53.319519 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:29:53.319489    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:30:00.134522 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.134476    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc"]
Jul 29 21:30:00.138097 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.138075    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5"]
Jul 29 21:30:00.138214 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.138189    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc"
Jul 29 21:30:00.140968 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.140941    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wpqrc\""
Jul 29 21:30:00.141881 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.141858    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5"
Jul 29 21:30:00.145024 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.144996    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc"]
Jul 29 21:30:00.152731 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.152709    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc"
Jul 29 21:30:00.160948 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.160920    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5"]
Jul 29 21:30:00.162374 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.162302    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5"
Jul 29 21:30:00.300177 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.300081    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc"]
Jul 29 21:30:00.302119 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:30:00.302085    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79725c7c_909a_4783_9648_728d467cbf24.slice/crio-7340d21b9e6592f5577928fdf323d9213368cbe4e35cd14b5385d8480123ff4c WatchSource:0}: Error finding container 7340d21b9e6592f5577928fdf323d9213368cbe4e35cd14b5385d8480123ff4c: Status 404 returned error can't find the container with id 7340d21b9e6592f5577928fdf323d9213368cbe4e35cd14b5385d8480123ff4c
Jul 29 21:30:00.324604 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.324580    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5"]
Jul 29 21:30:00.326297 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:30:00.326267    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d83dd6b_b73c_4ab7_bfac_fc4fba1126ad.slice/crio-254c79cc2104ea69605261e6f24f72c50657ae861cfc0644d46ee9252ccfd432 WatchSource:0}: Error finding container 254c79cc2104ea69605261e6f24f72c50657ae861cfc0644d46ee9252ccfd432: Status 404 returned error can't find the container with id 254c79cc2104ea69605261e6f24f72c50657ae861cfc0644d46ee9252ccfd432
Jul 29 21:30:00.378062 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.377988    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5" event={"ID":"3d83dd6b-b73c-4ab7-bfac-fc4fba1126ad","Type":"ContainerStarted","Data":"254c79cc2104ea69605261e6f24f72c50657ae861cfc0644d46ee9252ccfd432"}
Jul 29 21:30:00.379837 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:00.379813    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc" event={"ID":"79725c7c-909a-4783-9648-728d467cbf24","Type":"ContainerStarted","Data":"7340d21b9e6592f5577928fdf323d9213368cbe4e35cd14b5385d8480123ff4c"}
Jul 29 21:30:01.386279 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:01.386246    2585 generic.go:356] "Generic (PLEG): container finished" podID="3d83dd6b-b73c-4ab7-bfac-fc4fba1126ad" containerID="14d18e1e31f31b474c1f2e67ab923e5819e2c724a5466148b6b6d2443011f2de" exitCode=0
Jul 29 21:30:01.386687 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:01.386358    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5" event={"ID":"3d83dd6b-b73c-4ab7-bfac-fc4fba1126ad","Type":"ContainerDied","Data":"14d18e1e31f31b474c1f2e67ab923e5819e2c724a5466148b6b6d2443011f2de"}
Jul 29 21:30:01.387889 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:01.387868    2585 generic.go:356] "Generic (PLEG): container finished" podID="79725c7c-909a-4783-9648-728d467cbf24" containerID="18f6a42932dc6d722fd2352714da98501aeead2456b7975d5ce6ca6d77fd4c8f" exitCode=0
Jul 29 21:30:01.387982 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:01.387955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc" event={"ID":"79725c7c-909a-4783-9648-728d467cbf24","Type":"ContainerDied","Data":"18f6a42932dc6d722fd2352714da98501aeead2456b7975d5ce6ca6d77fd4c8f"}
Jul 29 21:30:02.553401 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:02.553375    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5"
Jul 29 21:30:02.579547 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:02.579506    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc"
Jul 29 21:30:03.398984 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:03.398959    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5"
Jul 29 21:30:03.398984 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:03.398975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5" event={"ID":"3d83dd6b-b73c-4ab7-bfac-fc4fba1126ad","Type":"ContainerDied","Data":"254c79cc2104ea69605261e6f24f72c50657ae861cfc0644d46ee9252ccfd432"}
Jul 29 21:30:03.399198 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:03.399022    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="254c79cc2104ea69605261e6f24f72c50657ae861cfc0644d46ee9252ccfd432"
Jul 29 21:30:03.400287 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:03.400266    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc"
Jul 29 21:30:03.400287 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:03.400268    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc" event={"ID":"79725c7c-909a-4783-9648-728d467cbf24","Type":"ContainerDied","Data":"7340d21b9e6592f5577928fdf323d9213368cbe4e35cd14b5385d8480123ff4c"}
Jul 29 21:30:03.400481 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:03.400314    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7340d21b9e6592f5577928fdf323d9213368cbe4e35cd14b5385d8480123ff4c"
Jul 29 21:30:10.230989 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:10.230952    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:25.643146 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.643112    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"]
Jul 29 21:30:25.643623 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.643318    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn" podUID="17a5ef05-e27a-4f42-9788-6b4734b923b0" containerName="maas-api" containerID="cri-o://b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579" gracePeriod=30
Jul 29 21:30:25.669262 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.669238    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc"]
Jul 29 21:30:25.675898 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.675873    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0109d4-29756010-vmpvc"]
Jul 29 21:30:25.677543 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.677506    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:25.753017 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.752985    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:25.825647 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.825581    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:25.894230 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.894177    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:30:25.959674 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.959647    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:25.990741 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.990714    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a5ef05-e27a-4f42-9788-6b4734b923b0-maas-api-tls\") pod \"17a5ef05-e27a-4f42-9788-6b4734b923b0\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") "
Jul 29 21:30:25.990866 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.990813    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5tgn\" (UniqueName: \"kubernetes.io/projected/17a5ef05-e27a-4f42-9788-6b4734b923b0-kube-api-access-h5tgn\") pod \"17a5ef05-e27a-4f42-9788-6b4734b923b0\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") "
Jul 29 21:30:25.990913 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.990865    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ef05-e27a-4f42-9788-6b4734b923b0-openshift-service-ca\") pod \"17a5ef05-e27a-4f42-9788-6b4734b923b0\" (UID: \"17a5ef05-e27a-4f42-9788-6b4734b923b0\") "
Jul 29 21:30:25.991254 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.991228    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a5ef05-e27a-4f42-9788-6b4734b923b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17a5ef05-e27a-4f42-9788-6b4734b923b0" (UID: "17a5ef05-e27a-4f42-9788-6b4734b923b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:30:25.992759 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.992736    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a5ef05-e27a-4f42-9788-6b4734b923b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17a5ef05-e27a-4f42-9788-6b4734b923b0" (UID: "17a5ef05-e27a-4f42-9788-6b4734b923b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:30:25.992883 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:25.992866    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a5ef05-e27a-4f42-9788-6b4734b923b0-kube-api-access-h5tgn" (OuterVolumeSpecName: "kube-api-access-h5tgn") pod "17a5ef05-e27a-4f42-9788-6b4734b923b0" (UID: "17a5ef05-e27a-4f42-9788-6b4734b923b0"). InnerVolumeSpecName "kube-api-access-h5tgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:30:26.092274 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.092226    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tgn\" (UniqueName: \"kubernetes.io/projected/17a5ef05-e27a-4f42-9788-6b4734b923b0-kube-api-access-h5tgn\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:30:26.092274 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.092271    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ef05-e27a-4f42-9788-6b4734b923b0-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:30:26.092274 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.092283    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a5ef05-e27a-4f42-9788-6b4734b923b0-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:30:26.262976 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.262905    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:26.515605 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.515515    2585 generic.go:356] "Generic (PLEG): container finished" podID="17a5ef05-e27a-4f42-9788-6b4734b923b0" containerID="b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579" exitCode=0
Jul 29 21:30:26.515605 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.515564    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"
Jul 29 21:30:26.515826 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.515609    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn" event={"ID":"17a5ef05-e27a-4f42-9788-6b4734b923b0","Type":"ContainerDied","Data":"b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579"}
Jul 29 21:30:26.515826 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.515665    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn" event={"ID":"17a5ef05-e27a-4f42-9788-6b4734b923b0","Type":"ContainerDied","Data":"6db28c3db4c1d7bf21eba003f51dd82d35d3f9a423ae94b8a2c9516e72d6f3d1"}
Jul 29 21:30:26.515826 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.515698    2585 scope.go:117] "RemoveContainer" containerID="b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579"
Jul 29 21:30:26.524826 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.524803    2585 scope.go:117] "RemoveContainer" containerID="b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579"
Jul 29 21:30:26.525075 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:30:26.525056    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579\": container with ID starting with b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579 not found: ID does not exist" containerID="b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579"
Jul 29 21:30:26.525185 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.525083    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579"} err="failed to get container status \"b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579\": rpc error: code = NotFound desc = could not find container \"b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579\": container with ID starting with b18cfe27bbcfbcce41aeef54633020d5a8ad0780b918a085419e04b3b588e579 not found: ID does not exist"
Jul 29 21:30:26.538141 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.538118    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"]
Jul 29 21:30:26.543518 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:26.543498    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0109d4-5647fc888f-2qmzn"]
Jul 29 21:30:27.524523 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:27.524487    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a5ef05-e27a-4f42-9788-6b4734b923b0" path="/var/lib/kubelet/pods/17a5ef05-e27a-4f42-9788-6b4734b923b0/volumes"
Jul 29 21:30:27.524936 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:27.524812    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79725c7c-909a-4783-9648-728d467cbf24" path="/var/lib/kubelet/pods/79725c7c-909a-4783-9648-728d467cbf24/volumes"
Jul 29 21:30:32.757113 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:32.757068    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:32.856792 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:32.856766    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:33.063824 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.063745    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:33.284128 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.284099    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:33.395683 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.395618    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"]
Jul 29 21:30:33.395983 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.395972    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d83dd6b-b73c-4ab7-bfac-fc4fba1126ad" containerName="cleanup"
Jul 29 21:30:33.396033 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.395985    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d83dd6b-b73c-4ab7-bfac-fc4fba1126ad" containerName="cleanup"
Jul 29 21:30:33.396033 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.396001    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79725c7c-909a-4783-9648-728d467cbf24" containerName="cleanup"
Jul 29 21:30:33.396033 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.396008    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79725c7c-909a-4783-9648-728d467cbf24" containerName="cleanup"
Jul 29 21:30:33.396033 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.396016    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a5ef05-e27a-4f42-9788-6b4734b923b0" containerName="maas-api"
Jul 29 21:30:33.396033 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.396021    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a5ef05-e27a-4f42-9788-6b4734b923b0" containerName="maas-api"
Jul 29 21:30:33.396205 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.396082    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="79725c7c-909a-4783-9648-728d467cbf24" containerName="cleanup"
Jul 29 21:30:33.396205 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.396090    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d83dd6b-b73c-4ab7-bfac-fc4fba1126ad" containerName="cleanup"
Jul 29 21:30:33.396205 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.396098    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a5ef05-e27a-4f42-9788-6b4734b923b0" containerName="maas-api"
Jul 29 21:30:33.400558 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.400536    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"
Jul 29 21:30:33.403219 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.403194    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wpqrc\""
Jul 29 21:30:33.410246 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.410226    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"]
Jul 29 21:30:33.557380 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.557325    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727a492-7859-4be5-8374-262bb4623641-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b\" (UID: \"b727a492-7859-4be5-8374-262bb4623641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"
Jul 29 21:30:33.658738 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.658670    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727a492-7859-4be5-8374-262bb4623641-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b\" (UID: \"b727a492-7859-4be5-8374-262bb4623641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"
Jul 29 21:30:33.659254 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.659234    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727a492-7859-4be5-8374-262bb4623641-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b\" (UID: \"b727a492-7859-4be5-8374-262bb4623641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"
Jul 29 21:30:33.711194 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.711162    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"
Jul 29 21:30:33.849249 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:33.849221    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"]
Jul 29 21:30:33.851603 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:30:33.851572    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb727a492_7859_4be5_8374_262bb4623641.slice/crio-a496ee135a1618e47dc1933ccf5a8ead1cd8082ac4113007f7d12b637c4018fc WatchSource:0}: Error finding container a496ee135a1618e47dc1933ccf5a8ead1cd8082ac4113007f7d12b637c4018fc: Status 404 returned error can't find the container with id a496ee135a1618e47dc1933ccf5a8ead1cd8082ac4113007f7d12b637c4018fc
Jul 29 21:30:34.556711 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:34.556689    2585 generic.go:356] "Generic (PLEG): container finished" podID="b727a492-7859-4be5-8374-262bb4623641" containerID="da9b8d48dd8c87855442297c069ff726770844be5b0c1838a5c795c15dd1ab3f" exitCode=0
Jul 29 21:30:34.556808 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:34.556730    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b" event={"ID":"b727a492-7859-4be5-8374-262bb4623641","Type":"ContainerDied","Data":"da9b8d48dd8c87855442297c069ff726770844be5b0c1838a5c795c15dd1ab3f"}
Jul 29 21:30:34.556808 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:34.556766    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b" event={"ID":"b727a492-7859-4be5-8374-262bb4623641","Type":"ContainerStarted","Data":"a496ee135a1618e47dc1933ccf5a8ead1cd8082ac4113007f7d12b637c4018fc"}
Jul 29 21:30:35.691117 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:35.691094    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"
Jul 29 21:30:35.778372 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:35.778326    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727a492-7859-4be5-8374-262bb4623641-maas-api-service-ca\") pod \"b727a492-7859-4be5-8374-262bb4623641\" (UID: \"b727a492-7859-4be5-8374-262bb4623641\") "
Jul 29 21:30:35.778641 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:35.778620    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b727a492-7859-4be5-8374-262bb4623641-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b727a492-7859-4be5-8374-262bb4623641" (UID: "b727a492-7859-4be5-8374-262bb4623641"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:30:35.879713 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:35.879653    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727a492-7859-4be5-8374-262bb4623641-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:30:36.567705 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:36.567675    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"
Jul 29 21:30:36.567841 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:36.567677    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b" event={"ID":"b727a492-7859-4be5-8374-262bb4623641","Type":"ContainerDied","Data":"a496ee135a1618e47dc1933ccf5a8ead1cd8082ac4113007f7d12b637c4018fc"}
Jul 29 21:30:36.567841 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:36.567802    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a496ee135a1618e47dc1933ccf5a8ead1cd8082ac4113007f7d12b637c4018fc"
Jul 29 21:30:43.899203 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:43.899164    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"]
Jul 29 21:30:43.899656 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:43.899469    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk" podUID="65415b0e-eb68-4178-be35-8d3685ccb03e" containerName="maas-api" containerID="cri-o://e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012" gracePeriod=30
Jul 29 21:30:43.926691 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:43.926665    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5"]
Jul 29 21:30:43.930964 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:43.930941    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dfd5ca-29756010-4lvt5"]
Jul 29 21:30:44.004729 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.004699    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:44.105528 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.105493    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:44.122773 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.122716    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"]
Jul 29 21:30:44.124741 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.124717    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfd5ca-psf8b"]
Jul 29 21:30:44.146097 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.146073    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:30:44.170360 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.170280    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:44.248453 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.248432    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65415b0e-eb68-4178-be35-8d3685ccb03e-openshift-service-ca\") pod \"65415b0e-eb68-4178-be35-8d3685ccb03e\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") "
Jul 29 21:30:44.248565 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.248461    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65415b0e-eb68-4178-be35-8d3685ccb03e-maas-api-tls\") pod \"65415b0e-eb68-4178-be35-8d3685ccb03e\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") "
Jul 29 21:30:44.248565 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.248513    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hwcc\" (UniqueName: \"kubernetes.io/projected/65415b0e-eb68-4178-be35-8d3685ccb03e-kube-api-access-2hwcc\") pod \"65415b0e-eb68-4178-be35-8d3685ccb03e\" (UID: \"65415b0e-eb68-4178-be35-8d3685ccb03e\") "
Jul 29 21:30:44.248848 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.248818    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65415b0e-eb68-4178-be35-8d3685ccb03e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65415b0e-eb68-4178-be35-8d3685ccb03e" (UID: "65415b0e-eb68-4178-be35-8d3685ccb03e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:30:44.250583 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.250564    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65415b0e-eb68-4178-be35-8d3685ccb03e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65415b0e-eb68-4178-be35-8d3685ccb03e" (UID: "65415b0e-eb68-4178-be35-8d3685ccb03e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:30:44.250658 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.250636    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65415b0e-eb68-4178-be35-8d3685ccb03e-kube-api-access-2hwcc" (OuterVolumeSpecName: "kube-api-access-2hwcc") pod "65415b0e-eb68-4178-be35-8d3685ccb03e" (UID: "65415b0e-eb68-4178-be35-8d3685ccb03e"). InnerVolumeSpecName "kube-api-access-2hwcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:30:44.349696 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.349675    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65415b0e-eb68-4178-be35-8d3685ccb03e-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:30:44.349803 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.349699    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65415b0e-eb68-4178-be35-8d3685ccb03e-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:30:44.349803 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.349714    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hwcc\" (UniqueName: \"kubernetes.io/projected/65415b0e-eb68-4178-be35-8d3685ccb03e-kube-api-access-2hwcc\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:30:44.503971 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.503945    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:44.605688 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.605661    2585 generic.go:356] "Generic (PLEG): container finished" podID="65415b0e-eb68-4178-be35-8d3685ccb03e" containerID="e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012" exitCode=0
Jul 29 21:30:44.605790 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.605716    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"
Jul 29 21:30:44.605790 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.605748    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk" event={"ID":"65415b0e-eb68-4178-be35-8d3685ccb03e","Type":"ContainerDied","Data":"e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012"}
Jul 29 21:30:44.605859 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.605827    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk" event={"ID":"65415b0e-eb68-4178-be35-8d3685ccb03e","Type":"ContainerDied","Data":"9b6c1112319125fd8c9761af59215ad06ebd606bf8f3699c1c54f34fcb043518"}
Jul 29 21:30:44.605859 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.605855    2585 scope.go:117] "RemoveContainer" containerID="e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012"
Jul 29 21:30:44.614517 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.614500    2585 scope.go:117] "RemoveContainer" containerID="e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012"
Jul 29 21:30:44.614786 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:30:44.614764    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012\": container with ID starting with e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012 not found: ID does not exist" containerID="e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012"
Jul 29 21:30:44.614834 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.614798    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012"} err="failed to get container status \"e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012\": rpc error: code = NotFound desc = could not find container \"e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012\": container with ID starting with e4dc96ef74c125e5ad7f673f7c79373657d62a4ad229ec9b61ce46878f553012 not found: ID does not exist"
Jul 29 21:30:44.627539 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.627507    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"]
Jul 29 21:30:44.632499 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:44.632472    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfd5ca-794d9d7575-7fnkk"]
Jul 29 21:30:45.525023 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:45.524993    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d83dd6b-b73c-4ab7-bfac-fc4fba1126ad" path="/var/lib/kubelet/pods/3d83dd6b-b73c-4ab7-bfac-fc4fba1126ad/volumes"
Jul 29 21:30:45.525403 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:45.525359    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65415b0e-eb68-4178-be35-8d3685ccb03e" path="/var/lib/kubelet/pods/65415b0e-eb68-4178-be35-8d3685ccb03e/volumes"
Jul 29 21:30:45.525672 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:45.525657    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b727a492-7859-4be5-8374-262bb4623641" path="/var/lib/kubelet/pods/b727a492-7859-4be5-8374-262bb4623641/volumes"
Jul 29 21:30:50.573204 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:50.573154    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:50.655920 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:50.655888    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:50.834371 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:50.834280    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:51.070778 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:51.070747    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:51.381130 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:51.381103    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:51.689248 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:51.689165    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:51.803710 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:51.803675    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:54.607776 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.607746    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:54.698416 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.698380    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"]
Jul 29 21:30:54.698778 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.698762    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b727a492-7859-4be5-8374-262bb4623641" containerName="revoke-keys"
Jul 29 21:30:54.698778 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.698777    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b727a492-7859-4be5-8374-262bb4623641" containerName="revoke-keys"
Jul 29 21:30:54.698921 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.698788    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65415b0e-eb68-4178-be35-8d3685ccb03e" containerName="maas-api"
Jul 29 21:30:54.698921 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.698794    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="65415b0e-eb68-4178-be35-8d3685ccb03e" containerName="maas-api"
Jul 29 21:30:54.698921 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.698857    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="65415b0e-eb68-4178-be35-8d3685ccb03e" containerName="maas-api"
Jul 29 21:30:54.698921 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.698866    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b727a492-7859-4be5-8374-262bb4623641" containerName="revoke-keys"
Jul 29 21:30:54.701028 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.700992    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:54.701881 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.701861    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:54.704930 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.704908    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-5f7d51-serving-cert\""
Jul 29 21:30:54.717098 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.717078    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"]
Jul 29 21:30:54.787039 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.787008    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:54.834270 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.834239    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:54.834270 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.834272    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:54.834485 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.834361    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sr65\" (UniqueName: \"kubernetes.io/projected/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-kube-api-access-8sr65\") pod \"maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:54.883886 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.883822    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:54.935159 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.935129    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sr65\" (UniqueName: \"kubernetes.io/projected/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-kube-api-access-8sr65\") pod \"maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:54.935289 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.935241    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:54.935289 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.935272    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:54.935891 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.935868    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:54.937863 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.937833    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:54.943706 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:54.943683    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sr65\" (UniqueName: \"kubernetes.io/projected/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-kube-api-access-8sr65\") pod \"maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:55.011970 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:55.011949    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:55.140213 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:55.140186    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"]
Jul 29 21:30:55.142314 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:30:55.142286    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b87f01_aeb3_4cc0_b8d3_d8a8f713c505.slice/crio-5470a5bee3a3031b1f3165bf1b4b9c8c35589ac0a9b40b11037a59337a0fff39 WatchSource:0}: Error finding container 5470a5bee3a3031b1f3165bf1b4b9c8c35589ac0a9b40b11037a59337a0fff39: Status 404 returned error can't find the container with id 5470a5bee3a3031b1f3165bf1b4b9c8c35589ac0a9b40b11037a59337a0fff39
Jul 29 21:30:55.378094 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:55.378059    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:55.481116 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:55.481078    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:30:55.666840 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:55.666808    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5" event={"ID":"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505","Type":"ContainerStarted","Data":"87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905"}
Jul 29 21:30:55.667195 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:55.666891    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5" event={"ID":"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505","Type":"ContainerStarted","Data":"5470a5bee3a3031b1f3165bf1b4b9c8c35589ac0a9b40b11037a59337a0fff39"}
Jul 29 21:30:55.667195 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:55.666920    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:30:55.685556 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:30:55.685500    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5" podStartSLOduration=1.2537801769999999 podStartE2EDuration="1.685484778s" totalImagesPullingTime="431.704601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:55.684577477 +0000 UTC m=+3982.728269661" watchObservedRunningTime="2026-07-29 21:30:55.685484778 +0000 UTC m=+3982.729176961"
Jul 29 21:31:01.677935 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:01.677908    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:31:19.573742 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:19.573710    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:19.939541 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:19.939461    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:20.244764 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:20.244733    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:20.604394 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:20.604302    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:22.821406 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:22.821372    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:22.923436 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:22.923405    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:23.034153 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.034122    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"]
Jul 29 21:31:23.037737 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.037715    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.040611 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.040587    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-eee583-serving-cert\""
Jul 29 21:31:23.050154 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.050131    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"]
Jul 29 21:31:23.084229 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.084170    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:23.151401 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.151368    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bstfz\" (UniqueName: \"kubernetes.io/projected/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-kube-api-access-bstfz\") pod \"maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.151533 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.151423    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-openshift-service-ca\") pod \"maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.151583 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.151555    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-maas-api-tls\") pod \"maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.252919 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.252889    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-maas-api-tls\") pod \"maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.253093 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.252963    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bstfz\" (UniqueName: \"kubernetes.io/projected/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-kube-api-access-bstfz\") pod \"maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.253093 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.253003    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-openshift-service-ca\") pod \"maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.253688 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.253665    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-openshift-service-ca\") pod \"maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.255393 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.255372    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-maas-api-tls\") pod \"maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.263076 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.263052    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:23.264304 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.264284    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bstfz\" (UniqueName: \"kubernetes.io/projected/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-kube-api-access-bstfz\") pod \"maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.350652 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.350593    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:23.485077 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.484974    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"]
Jul 29 21:31:23.487175 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:31:23.487148    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6db15fd_e9b8_4c5b_a6e2_4a658ccaeb57.slice/crio-676ecdcd50f126f3812321822f031164f59e479dfe71ab536ae1f15320c5fbbb WatchSource:0}: Error finding container 676ecdcd50f126f3812321822f031164f59e479dfe71ab536ae1f15320c5fbbb: Status 404 returned error can't find the container with id 676ecdcd50f126f3812321822f031164f59e479dfe71ab536ae1f15320c5fbbb
Jul 29 21:31:23.488586 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.488565    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:31:23.597302 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.597257    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:23.693878 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.693804    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:23.811637 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:23.811604    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5" event={"ID":"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57","Type":"ContainerStarted","Data":"676ecdcd50f126f3812321822f031164f59e479dfe71ab536ae1f15320c5fbbb"}
Jul 29 21:31:24.818872 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:24.818830    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5" event={"ID":"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57","Type":"ContainerStarted","Data":"d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51"}
Jul 29 21:31:24.819300 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:24.818929    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:30.829493 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:30.829459    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:31:30.849048 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:30.848988    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5" podStartSLOduration=7.432407756 podStartE2EDuration="7.848972119s" totalImagesPullingTime="416.564363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:31:24.837647442 +0000 UTC m=+4011.881339625" watchObservedRunningTime="2026-07-29 21:31:30.848972119 +0000 UTC m=+4017.892664374"
Jul 29 21:31:39.425139 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:39.425075    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:31:47.762694 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:31:47.762662    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:32:02.234732 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:02.234647    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:32:09.839002 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:09.838961    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:32:09.941883 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:09.941853    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:32:19.026068 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.026030    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"]
Jul 29 21:32:19.029911 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.029890    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.033913 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.033889    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"auth-test-model-eee583-kserve-self-signed-certs\""
Jul 29 21:32:19.034090 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.033984    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"openshift-service-ca.crt\""
Jul 29 21:32:19.034090 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.034015    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"default-dockercfg-x64vb\""
Jul 29 21:32:19.034090 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.034022    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"kube-root-ca.crt\""
Jul 29 21:32:19.038452 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.038428    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"]
Jul 29 21:32:19.049050 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.049023    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:32:19.102616 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.102588    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a3537c-e34f-4e11-98a0-df3686758d4e-tls-certs\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.102722 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.102626    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-home\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.102722 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.102646    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-model-cache\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.102722 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.102667    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-tmp-dir\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.102863 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.102759    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmfw\" (UniqueName: \"kubernetes.io/projected/23a3537c-e34f-4e11-98a0-df3686758d4e-kube-api-access-fgmfw\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.102918 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.102855    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-dshm\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.203363 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.203317    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmfw\" (UniqueName: \"kubernetes.io/projected/23a3537c-e34f-4e11-98a0-df3686758d4e-kube-api-access-fgmfw\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.203455 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.203389    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-dshm\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.203455 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.203415    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a3537c-e34f-4e11-98a0-df3686758d4e-tls-certs\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.203455 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.203441    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-home\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.203616 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.203458    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-model-cache\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.203616 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.203593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-tmp-dir\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.203785 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.203772    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-model-cache\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.203882 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.203863    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-home\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.203934 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.203920    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-tmp-dir\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.205748 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.205728    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-dshm\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.206063 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.206046    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a3537c-e34f-4e11-98a0-df3686758d4e-tls-certs\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.211763 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.211740    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmfw\" (UniqueName: \"kubernetes.io/projected/23a3537c-e34f-4e11-98a0-df3686758d4e-kube-api-access-fgmfw\") pod \"auth-test-model-eee583-kserve-7d76fc6cf-n8m9p\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") " pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.341323 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.341268    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:19.468712 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:19.468682    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"]
Jul 29 21:32:19.470162 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:32:19.470140    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23a3537c_e34f_4e11_98a0_df3686758d4e.slice/crio-cbab0551f0ffa79e23f851da81ee60ffe099f51e9495485e0e3adde581310ef8 WatchSource:0}: Error finding container cbab0551f0ffa79e23f851da81ee60ffe099f51e9495485e0e3adde581310ef8: Status 404 returned error can't find the container with id cbab0551f0ffa79e23f851da81ee60ffe099f51e9495485e0e3adde581310ef8
Jul 29 21:32:20.103393 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:20.103359    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p" event={"ID":"23a3537c-e34f-4e11-98a0-df3686758d4e","Type":"ContainerStarted","Data":"3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec"}
Jul 29 21:32:20.103791 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:20.103404    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p" event={"ID":"23a3537c-e34f-4e11-98a0-df3686758d4e","Type":"ContainerStarted","Data":"cbab0551f0ffa79e23f851da81ee60ffe099f51e9495485e0e3adde581310ef8"}
Jul 29 21:32:20.122034 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:20.121983    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p" podStartSLOduration=1.121967392 podStartE2EDuration="1.121967392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:32:20.119908673 +0000 UTC m=+4067.163600858" watchObservedRunningTime="2026-07-29 21:32:20.121967392 +0000 UTC m=+4067.165659642"
Jul 29 21:32:29.341841 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:29.341802    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:29.342285 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:29.341855    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:29.354795 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:29.354764    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:32:30.164364 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:32:30.164318    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:33:10.630899 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:33:10.630820    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:33:10.898778 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:33:10.898691    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:34:33.806981 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:34:33.806950    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:34:33.812644 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:34:33.812619    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:34:51.491620 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:34:51.491576    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:35:13.795446 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:35:13.795361    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:35:33.983983 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:35:33.983951    2585 scope.go:117] "RemoveContainer" containerID="df34fa9e5e6789d08d23713d400b5a7de9c31e355341bb24ffd41e3aaf8ce6cb"
Jul 29 21:36:34.006733 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:36:34.006649    2585 scope.go:117] "RemoveContainer" containerID="14d18e1e31f31b474c1f2e67ab923e5819e2c724a5466148b6b6d2443011f2de"
Jul 29 21:36:34.018888 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:36:34.018867    2585 scope.go:117] "RemoveContainer" containerID="18f6a42932dc6d722fd2352714da98501aeead2456b7975d5ce6ca6d77fd4c8f"
Jul 29 21:36:34.026915 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:36:34.026897    2585 scope.go:117] "RemoveContainer" containerID="da9b8d48dd8c87855442297c069ff726770844be5b0c1838a5c795c15dd1ab3f"
Jul 29 21:37:08.893232 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:08.893198    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:37:09.766128 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:09.766098    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"]
Jul 29 21:37:10.551817 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:10.551786    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:37:24.784855 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:24.784795    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p" podUID="23a3537c-e34f-4e11-98a0-df3686758d4e" containerName="main" containerID="cri-o://3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec" gracePeriod=45
Jul 29 21:37:25.048375 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.048301    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:37:25.114244 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114213    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgmfw\" (UniqueName: \"kubernetes.io/projected/23a3537c-e34f-4e11-98a0-df3686758d4e-kube-api-access-fgmfw\") pod \"23a3537c-e34f-4e11-98a0-df3686758d4e\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") "
Jul 29 21:37:25.114414 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114313    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-home\") pod \"23a3537c-e34f-4e11-98a0-df3686758d4e\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") "
Jul 29 21:37:25.114414 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114364    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-model-cache\") pod \"23a3537c-e34f-4e11-98a0-df3686758d4e\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") "
Jul 29 21:37:25.114414 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114402    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-tmp-dir\") pod \"23a3537c-e34f-4e11-98a0-df3686758d4e\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") "
Jul 29 21:37:25.114592 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114434    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a3537c-e34f-4e11-98a0-df3686758d4e-tls-certs\") pod \"23a3537c-e34f-4e11-98a0-df3686758d4e\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") "
Jul 29 21:37:25.114592 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114461    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-dshm\") pod \"23a3537c-e34f-4e11-98a0-df3686758d4e\" (UID: \"23a3537c-e34f-4e11-98a0-df3686758d4e\") "
Jul 29 21:37:25.114699 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114595    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-home" (OuterVolumeSpecName: "home") pod "23a3537c-e34f-4e11-98a0-df3686758d4e" (UID: "23a3537c-e34f-4e11-98a0-df3686758d4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:25.114699 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114644    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "23a3537c-e34f-4e11-98a0-df3686758d4e" (UID: "23a3537c-e34f-4e11-98a0-df3686758d4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:25.114815 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114690    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23a3537c-e34f-4e11-98a0-df3686758d4e" (UID: "23a3537c-e34f-4e11-98a0-df3686758d4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:25.114895 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114874    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:37:25.114953 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114904    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:37:25.114953 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.114945    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:37:25.116498 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.116468    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-dshm" (OuterVolumeSpecName: "dshm") pod "23a3537c-e34f-4e11-98a0-df3686758d4e" (UID: "23a3537c-e34f-4e11-98a0-df3686758d4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:25.116611 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.116513    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a3537c-e34f-4e11-98a0-df3686758d4e-kube-api-access-fgmfw" (OuterVolumeSpecName: "kube-api-access-fgmfw") pod "23a3537c-e34f-4e11-98a0-df3686758d4e" (UID: "23a3537c-e34f-4e11-98a0-df3686758d4e"). InnerVolumeSpecName "kube-api-access-fgmfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:37:25.116611 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.116522    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a3537c-e34f-4e11-98a0-df3686758d4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23a3537c-e34f-4e11-98a0-df3686758d4e" (UID: "23a3537c-e34f-4e11-98a0-df3686758d4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:37:25.216040 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.216000    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a3537c-e34f-4e11-98a0-df3686758d4e-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:37:25.216040 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.216037    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a3537c-e34f-4e11-98a0-df3686758d4e-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:37:25.216040 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.216047    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgmfw\" (UniqueName: \"kubernetes.io/projected/23a3537c-e34f-4e11-98a0-df3686758d4e-kube-api-access-fgmfw\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:37:25.623730 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.623698    2585 generic.go:356] "Generic (PLEG): container finished" podID="23a3537c-e34f-4e11-98a0-df3686758d4e" containerID="3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec" exitCode=0
Jul 29 21:37:25.623901 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.623738    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"
Jul 29 21:37:25.623901 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.623737    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p" event={"ID":"23a3537c-e34f-4e11-98a0-df3686758d4e","Type":"ContainerDied","Data":"3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec"}
Jul 29 21:37:25.623901 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.623829    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p" event={"ID":"23a3537c-e34f-4e11-98a0-df3686758d4e","Type":"ContainerDied","Data":"cbab0551f0ffa79e23f851da81ee60ffe099f51e9495485e0e3adde581310ef8"}
Jul 29 21:37:25.623901 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.623852    2585 scope.go:117] "RemoveContainer" containerID="3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec"
Jul 29 21:37:25.632610 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.632592    2585 scope.go:117] "RemoveContainer" containerID="3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec"
Jul 29 21:37:25.632859 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:37:25.632838    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec\": container with ID starting with 3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec not found: ID does not exist" containerID="3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec"
Jul 29 21:37:25.632907 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.632868    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec"} err="failed to get container status \"3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec\": rpc error: code = NotFound desc = could not find container \"3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec\": container with ID starting with 3a6cd1708a8cad1cb242c9a110d205634c7b1bb552f68ea0174f018227d1a2ec not found: ID does not exist"
Jul 29 21:37:25.645260 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.645225    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"]
Jul 29 21:37:25.647784 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:25.647759    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/auth-test-model-eee583-kserve-7d76fc6cf-n8m9p"]
Jul 29 21:37:27.524562 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:27.524532    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23a3537c-e34f-4e11-98a0-df3686758d4e" path="/var/lib/kubelet/pods/23a3537c-e34f-4e11-98a0-df3686758d4e/volumes"
Jul 29 21:37:30.066798 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.066765    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"]
Jul 29 21:37:30.067156 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.067130    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a3537c-e34f-4e11-98a0-df3686758d4e" containerName="main"
Jul 29 21:37:30.067156 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.067141    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a3537c-e34f-4e11-98a0-df3686758d4e" containerName="main"
Jul 29 21:37:30.067229 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.067213    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a3537c-e34f-4e11-98a0-df3686758d4e" containerName="main"
Jul 29 21:37:30.071430 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.071413    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.075237 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.075209    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5f7d51\"/\"default-dockercfg-467pw\""
Jul 29 21:37:30.076217 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.076189    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5f7d51\"/\"kube-root-ca.crt\""
Jul 29 21:37:30.076362 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.076322    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5f7d51\"/\"openshift-service-ca.crt\""
Jul 29 21:37:30.076627 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.076600    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5f7d51\"/\"sub-test-model-5f7d51-kserve-self-signed-certs\""
Jul 29 21:37:30.098479 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.098455    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"]
Jul 29 21:37:30.147640 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.147619    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:37:30.156213 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.156192    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-model-cache\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.156297 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.156225    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmst\" (UniqueName: \"kubernetes.io/projected/92339730-c133-4416-922a-389abe830443-kube-api-access-5vmst\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.156297 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.156285    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-dshm\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.156406 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.156349    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92339730-c133-4416-922a-389abe830443-tls-certs\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.156406 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.156378    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-home\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.156406 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.156398    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-tmp-dir\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.256996 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.256966    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-dshm\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.257130 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.257020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92339730-c133-4416-922a-389abe830443-tls-certs\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.257130 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.257057    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-home\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.257130 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.257083    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-tmp-dir\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.257298 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.257149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-model-cache\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.257298 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.257180    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmst\" (UniqueName: \"kubernetes.io/projected/92339730-c133-4416-922a-389abe830443-kube-api-access-5vmst\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.257490 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.257464    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-home\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.257597 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.257528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-model-cache\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.257597 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.257544    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-tmp-dir\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.259379 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.259356    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-dshm\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.259659 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.259640    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92339730-c133-4416-922a-389abe830443-tls-certs\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.270766 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.270745    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmst\" (UniqueName: \"kubernetes.io/projected/92339730-c133-4416-922a-389abe830443-kube-api-access-5vmst\") pod \"sub-test-model-5f7d51-kserve-fd847c947-4lh7c\" (UID: \"92339730-c133-4416-922a-389abe830443\") " pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.386189 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.386130    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:30.527141 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.527109    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"]
Jul 29 21:37:30.530345 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:37:30.530298    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92339730_c133_4416_922a_389abe830443.slice/crio-7557ce8a29cd0e6d1178a57f517077b54133f7762853bacee0ce91a92cae7940 WatchSource:0}: Error finding container 7557ce8a29cd0e6d1178a57f517077b54133f7762853bacee0ce91a92cae7940: Status 404 returned error can't find the container with id 7557ce8a29cd0e6d1178a57f517077b54133f7762853bacee0ce91a92cae7940
Jul 29 21:37:30.649148 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.649069    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c" event={"ID":"92339730-c133-4416-922a-389abe830443","Type":"ContainerStarted","Data":"a83d9831170af3d9117ecb2dab405f01dd244dbf45f2338ce4e995a10d3f541c"}
Jul 29 21:37:30.649148 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.649110    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c" event={"ID":"92339730-c133-4416-922a-389abe830443","Type":"ContainerStarted","Data":"7557ce8a29cd0e6d1178a57f517077b54133f7762853bacee0ce91a92cae7940"}
Jul 29 21:37:30.692049 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:30.691997    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c" podStartSLOduration=0.691983126 podStartE2EDuration="691.983126ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:37:30.690326561 +0000 UTC m=+4377.734018745" watchObservedRunningTime="2026-07-29 21:37:30.691983126 +0000 UTC m=+4377.735675308"
Jul 29 21:37:40.386724 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:40.386682    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:40.386724 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:40.386733    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:40.399189 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:40.399161    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:40.706494 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:40.706430    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:37:54.967259 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:54.967228    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:37:55.046654 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:37:55.046621    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:38:07.785526 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:07.785447    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:38:07.849031 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:07.849007    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:38:08.932601 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:08.932570    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"]
Jul 29 21:38:08.936146 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:08.936125    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:08.940436 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:08.940412    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"default-dockercfg-x64vb\""
Jul 29 21:38:08.940560 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:08.940446    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"kube-root-ca.crt\""
Jul 29 21:38:08.940670 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:08.940654    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"sub-test-model-eee583-kserve-self-signed-certs\""
Jul 29 21:38:08.940758 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:08.940744    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"openshift-service-ca.crt\""
Jul 29 21:38:08.947227 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:08.947205    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"]
Jul 29 21:38:08.963460 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:08.963441    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:38:09.075043 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.075020    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-home\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.075140 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.075062    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwx9d\" (UniqueName: \"kubernetes.io/projected/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-kube-api-access-fwx9d\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.075140 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.075083    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-model-cache\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.075255 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.075211    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-dshm\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.075255 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.075244    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tmp-dir\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.075403 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.075309    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tls-certs\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.175851 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.175827    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-dshm\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.175950 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.175855    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tmp-dir\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.175950 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.175880    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tls-certs\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.176064 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.175982    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-home\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.176064 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.176039    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwx9d\" (UniqueName: \"kubernetes.io/projected/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-kube-api-access-fwx9d\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.176170 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.176069    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-model-cache\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.176170 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.176134    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tmp-dir\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.176367 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.176321    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-home\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.176442 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.176426    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-model-cache\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.177985 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.177964    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-dshm\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.178258 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.178241    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tls-certs\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.185668 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.185617    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwx9d\" (UniqueName: \"kubernetes.io/projected/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-kube-api-access-fwx9d\") pod \"sub-test-model-eee583-kserve-74d7b4c677-5h4gh\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") " pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.247967 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.247944    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:09.377657 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.377624    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"]
Jul 29 21:38:09.379574 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:38:09.379545    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb3902d_4a91_4ef5_bebe_81e7fba65a99.slice/crio-eae30111c01279fc1da6dd961f891df202fc617317f8abc253a2a2f17ef0002c WatchSource:0}: Error finding container eae30111c01279fc1da6dd961f891df202fc617317f8abc253a2a2f17ef0002c: Status 404 returned error can't find the container with id eae30111c01279fc1da6dd961f891df202fc617317f8abc253a2a2f17ef0002c
Jul 29 21:38:09.836908 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.836871    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh" event={"ID":"1fb3902d-4a91-4ef5-bebe-81e7fba65a99","Type":"ContainerStarted","Data":"246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d"}
Jul 29 21:38:09.837073 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.836922    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh" event={"ID":"1fb3902d-4a91-4ef5-bebe-81e7fba65a99","Type":"ContainerStarted","Data":"eae30111c01279fc1da6dd961f891df202fc617317f8abc253a2a2f17ef0002c"}
Jul 29 21:38:09.857449 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:09.857403    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh" podStartSLOduration=1.857388378 podStartE2EDuration="1.857388378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:38:09.855455267 +0000 UTC m=+4416.899147464" watchObservedRunningTime="2026-07-29 21:38:09.857388378 +0000 UTC m=+4416.901080561"
Jul 29 21:38:19.248720 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:19.248679    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:19.249146 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:19.248737    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:19.261274 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:19.261243    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:19.894547 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:19.894513    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:38:47.052751 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:47.052717    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:38:47.117518 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:47.117488    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:38:49.877639 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:38:49.877597    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:39:33.840112 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:39:33.840026    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:39:33.846213 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:39:33.846190    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:40:47.525683 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:40:47.525654    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"]
Jul 29 21:40:47.569762 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:40:47.569733    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:40:48.555259 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:40:48.555223    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"]
Jul 29 21:40:48.583444 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:40:48.583415    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:40:58.039940 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:40:58.039904    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:41:02.544409 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.544354    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c" podUID="92339730-c133-4416-922a-389abe830443" containerName="main" containerID="cri-o://a83d9831170af3d9117ecb2dab405f01dd244dbf45f2338ce4e995a10d3f541c" gracePeriod=45
Jul 29 21:41:02.676528 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.676500    2585 generic.go:356] "Generic (PLEG): container finished" podID="92339730-c133-4416-922a-389abe830443" containerID="a83d9831170af3d9117ecb2dab405f01dd244dbf45f2338ce4e995a10d3f541c" exitCode=0
Jul 29 21:41:02.676649 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.676575    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c" event={"ID":"92339730-c133-4416-922a-389abe830443","Type":"ContainerDied","Data":"a83d9831170af3d9117ecb2dab405f01dd244dbf45f2338ce4e995a10d3f541c"}
Jul 29 21:41:02.795715 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.795652    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:41:02.936349 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936303    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-tmp-dir\") pod \"92339730-c133-4416-922a-389abe830443\" (UID: \"92339730-c133-4416-922a-389abe830443\") "
Jul 29 21:41:02.936486 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936362    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-home\") pod \"92339730-c133-4416-922a-389abe830443\" (UID: \"92339730-c133-4416-922a-389abe830443\") "
Jul 29 21:41:02.936486 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936409    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-model-cache\") pod \"92339730-c133-4416-922a-389abe830443\" (UID: \"92339730-c133-4416-922a-389abe830443\") "
Jul 29 21:41:02.936486 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936426    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-dshm\") pod \"92339730-c133-4416-922a-389abe830443\" (UID: \"92339730-c133-4416-922a-389abe830443\") "
Jul 29 21:41:02.936486 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936469    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vmst\" (UniqueName: \"kubernetes.io/projected/92339730-c133-4416-922a-389abe830443-kube-api-access-5vmst\") pod \"92339730-c133-4416-922a-389abe830443\" (UID: \"92339730-c133-4416-922a-389abe830443\") "
Jul 29 21:41:02.936706 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936567    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92339730-c133-4416-922a-389abe830443-tls-certs\") pod \"92339730-c133-4416-922a-389abe830443\" (UID: \"92339730-c133-4416-922a-389abe830443\") "
Jul 29 21:41:02.936706 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936607    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92339730-c133-4416-922a-389abe830443" (UID: "92339730-c133-4416-922a-389abe830443"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:02.936706 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936692    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-model-cache" (OuterVolumeSpecName: "model-cache") pod "92339730-c133-4416-922a-389abe830443" (UID: "92339730-c133-4416-922a-389abe830443"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:02.936804 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936705    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-home" (OuterVolumeSpecName: "home") pod "92339730-c133-4416-922a-389abe830443" (UID: "92339730-c133-4416-922a-389abe830443"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:02.936934 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936917    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:02.936994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936936    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:02.936994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.936945    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:02.938538 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.938509    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-dshm" (OuterVolumeSpecName: "dshm") pod "92339730-c133-4416-922a-389abe830443" (UID: "92339730-c133-4416-922a-389abe830443"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:02.938626 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.938573    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92339730-c133-4416-922a-389abe830443-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92339730-c133-4416-922a-389abe830443" (UID: "92339730-c133-4416-922a-389abe830443"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:41:02.938626 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:02.938601    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92339730-c133-4416-922a-389abe830443-kube-api-access-5vmst" (OuterVolumeSpecName: "kube-api-access-5vmst") pod "92339730-c133-4416-922a-389abe830443" (UID: "92339730-c133-4416-922a-389abe830443"). InnerVolumeSpecName "kube-api-access-5vmst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:41:03.037995 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.037974    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92339730-c133-4416-922a-389abe830443-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:03.038088 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.038000    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmst\" (UniqueName: \"kubernetes.io/projected/92339730-c133-4416-922a-389abe830443-kube-api-access-5vmst\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:03.038088 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.038012    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92339730-c133-4416-922a-389abe830443-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:03.573726 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.573676    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh" podUID="1fb3902d-4a91-4ef5-bebe-81e7fba65a99" containerName="main" containerID="cri-o://246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d" gracePeriod=45
Jul 29 21:41:03.683122 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.683101    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"
Jul 29 21:41:03.683219 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.683126    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c" event={"ID":"92339730-c133-4416-922a-389abe830443","Type":"ContainerDied","Data":"7557ce8a29cd0e6d1178a57f517077b54133f7762853bacee0ce91a92cae7940"}
Jul 29 21:41:03.683219 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.683193    2585 scope.go:117] "RemoveContainer" containerID="a83d9831170af3d9117ecb2dab405f01dd244dbf45f2338ce4e995a10d3f541c"
Jul 29 21:41:03.701493 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.701464    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"]
Jul 29 21:41:03.707658 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.707627    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5f7d51/sub-test-model-5f7d51-kserve-fd847c947-4lh7c"]
Jul 29 21:41:03.833886 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.833862    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:41:03.945849 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.945827    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-model-cache\") pod \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") "
Jul 29 21:41:03.945970 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.945914    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-home\") pod \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") "
Jul 29 21:41:03.945970 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.945957    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tmp-dir\") pod \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") "
Jul 29 21:41:03.946089 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.945981    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-dshm\") pod \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") "
Jul 29 21:41:03.946089 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.946036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwx9d\" (UniqueName: \"kubernetes.io/projected/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-kube-api-access-fwx9d\") pod \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") "
Jul 29 21:41:03.946089 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.946066    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tls-certs\") pod \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\" (UID: \"1fb3902d-4a91-4ef5-bebe-81e7fba65a99\") "
Jul 29 21:41:03.946297 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.946094    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fb3902d-4a91-4ef5-bebe-81e7fba65a99" (UID: "1fb3902d-4a91-4ef5-bebe-81e7fba65a99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:03.946297 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.946213    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-home" (OuterVolumeSpecName: "home") pod "1fb3902d-4a91-4ef5-bebe-81e7fba65a99" (UID: "1fb3902d-4a91-4ef5-bebe-81e7fba65a99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:03.946297 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.946226    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fb3902d-4a91-4ef5-bebe-81e7fba65a99" (UID: "1fb3902d-4a91-4ef5-bebe-81e7fba65a99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:03.946458 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.946441    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:03.946498 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.946480    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:03.946498 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.946490    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:03.948176 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.948146    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-kube-api-access-fwx9d" (OuterVolumeSpecName: "kube-api-access-fwx9d") pod "1fb3902d-4a91-4ef5-bebe-81e7fba65a99" (UID: "1fb3902d-4a91-4ef5-bebe-81e7fba65a99"). InnerVolumeSpecName "kube-api-access-fwx9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:41:03.948280 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.948230    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-dshm" (OuterVolumeSpecName: "dshm") pod "1fb3902d-4a91-4ef5-bebe-81e7fba65a99" (UID: "1fb3902d-4a91-4ef5-bebe-81e7fba65a99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:03.948354 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:03.948275    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fb3902d-4a91-4ef5-bebe-81e7fba65a99" (UID: "1fb3902d-4a91-4ef5-bebe-81e7fba65a99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:41:04.047788 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.047763    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwx9d\" (UniqueName: \"kubernetes.io/projected/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-kube-api-access-fwx9d\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:04.047788 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.047786    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:04.047917 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.047795    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3902d-4a91-4ef5-bebe-81e7fba65a99-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:41:04.689361 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.689306    2585 generic.go:356] "Generic (PLEG): container finished" podID="1fb3902d-4a91-4ef5-bebe-81e7fba65a99" containerID="246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d" exitCode=0
Jul 29 21:41:04.689788 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.689368    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"
Jul 29 21:41:04.689788 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.689393    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh" event={"ID":"1fb3902d-4a91-4ef5-bebe-81e7fba65a99","Type":"ContainerDied","Data":"246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d"}
Jul 29 21:41:04.689788 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.689437    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh" event={"ID":"1fb3902d-4a91-4ef5-bebe-81e7fba65a99","Type":"ContainerDied","Data":"eae30111c01279fc1da6dd961f891df202fc617317f8abc253a2a2f17ef0002c"}
Jul 29 21:41:04.689788 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.689463    2585 scope.go:117] "RemoveContainer" containerID="246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d"
Jul 29 21:41:04.701139 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.701126    2585 scope.go:117] "RemoveContainer" containerID="246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d"
Jul 29 21:41:04.701374 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:41:04.701352    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d\": container with ID starting with 246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d not found: ID does not exist" containerID="246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d"
Jul 29 21:41:04.701489 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.701381    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d"} err="failed to get container status \"246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d\": rpc error: code = NotFound desc = could not find container \"246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d\": container with ID starting with 246f679b052fa4575caffd6eaa1af9f01e533de55521baa134ef4394f98bc59d not found: ID does not exist"
Jul 29 21:41:04.713723 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.713693    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"]
Jul 29 21:41:04.720226 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:04.720202    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/sub-test-model-eee583-kserve-74d7b4c677-5h4gh"]
Jul 29 21:41:05.525780 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:05.525751    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fb3902d-4a91-4ef5-bebe-81e7fba65a99" path="/var/lib/kubelet/pods/1fb3902d-4a91-4ef5-bebe-81e7fba65a99/volumes"
Jul 29 21:41:05.526158 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:05.526143    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92339730-c133-4416-922a-389abe830443" path="/var/lib/kubelet/pods/92339730-c133-4416-922a-389abe830443/volumes"
Jul 29 21:41:36.667375 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.667309    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"]
Jul 29 21:41:36.667781 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.667740    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb3902d-4a91-4ef5-bebe-81e7fba65a99" containerName="main"
Jul 29 21:41:36.667781 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.667753    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb3902d-4a91-4ef5-bebe-81e7fba65a99" containerName="main"
Jul 29 21:41:36.667781 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.667775    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92339730-c133-4416-922a-389abe830443" containerName="main"
Jul 29 21:41:36.667781 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.667780    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="92339730-c133-4416-922a-389abe830443" containerName="main"
Jul 29 21:41:36.667921 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.667853    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="92339730-c133-4416-922a-389abe830443" containerName="main"
Jul 29 21:41:36.667921 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.667864    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb3902d-4a91-4ef5-bebe-81e7fba65a99" containerName="main"
Jul 29 21:41:36.672651 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.672628    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.680491 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.680457    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"kube-root-ca.crt\""
Jul 29 21:41:36.681761 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.681740    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"default-dockercfg-x64vb\""
Jul 29 21:41:36.681891 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.681783    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"openshift-service-ca.crt\""
Jul 29 21:41:36.681966 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.681899    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-eee583\"/\"rate-test-model-eee583-kserve-self-signed-certs\""
Jul 29 21:41:36.702102 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.702078    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"]
Jul 29 21:41:36.722788 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.722760    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-dshm\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.722886 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.722854    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tls-certs\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.722950 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.722894    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chcxj\" (UniqueName: \"kubernetes.io/projected/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-kube-api-access-chcxj\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.722950 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.722913    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-home\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.723100 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.723076    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tmp-dir\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.723171 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.723125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-model-cache\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.774618 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.774593    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:41:36.823965 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.823941    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-model-cache\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.824074 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.823973    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-dshm\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.824074 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.824018    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tls-certs\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.824074 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.824042    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chcxj\" (UniqueName: \"kubernetes.io/projected/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-kube-api-access-chcxj\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.824074 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.824057    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-home\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.824295 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.824213    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tmp-dir\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.824450 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.824429    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-home\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.824532 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.824425    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-model-cache\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.824579 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.824548    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tmp-dir\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.826230 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.826212    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-dshm\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.826625 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.826605    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tls-certs\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.837728 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.837708    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chcxj\" (UniqueName: \"kubernetes.io/projected/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-kube-api-access-chcxj\") pod \"rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") " pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:36.983167 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:36.983144    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:37.118685 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:37.118661    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"]
Jul 29 21:41:37.120765 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:41:37.120736    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfbad70c_04c5_4d1d_a3e7_0a4ce574252f.slice/crio-612657ebeda2b3a6af306d884b1f1cfd00fb7cab704aebb6bdc6bb956beb6f34 WatchSource:0}: Error finding container 612657ebeda2b3a6af306d884b1f1cfd00fb7cab704aebb6bdc6bb956beb6f34: Status 404 returned error can't find the container with id 612657ebeda2b3a6af306d884b1f1cfd00fb7cab704aebb6bdc6bb956beb6f34
Jul 29 21:41:37.852016 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:37.851976    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8" event={"ID":"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f","Type":"ContainerStarted","Data":"fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5"}
Jul 29 21:41:37.852016 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:37.852018    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8" event={"ID":"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f","Type":"ContainerStarted","Data":"612657ebeda2b3a6af306d884b1f1cfd00fb7cab704aebb6bdc6bb956beb6f34"}
Jul 29 21:41:37.873809 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:37.873760    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8" podStartSLOduration=1.873745626 podStartE2EDuration="1.873745626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:41:37.872775879 +0000 UTC m=+4624.916468062" watchObservedRunningTime="2026-07-29 21:41:37.873745626 +0000 UTC m=+4624.917437809"
Jul 29 21:41:46.983750 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:46.983709    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:46.984197 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:46.983797    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:46.996339 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:46.996315    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:41:47.911840 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:41:47.911812    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:42:21.760224 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:42:21.760189    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:42:57.434842 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:42:57.434808    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:42:58.324040 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:42:58.324006    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"]
Jul 29 21:42:58.342204 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:42:58.342149    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:08.893701 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:08.893665    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"]
Jul 29 21:43:08.894160 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:08.893925    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5" podUID="06b87f01-aeb3-4cc0-b8d3-d8a8f713c505" containerName="maas-api" containerID="cri-o://87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905" gracePeriod=30
Jul 29 21:43:08.925676 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:08.925656    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:09.025942 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.025914    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:09.110371 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.110310    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:09.138662 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.138642    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:43:09.216862 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.216837    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sr65\" (UniqueName: \"kubernetes.io/projected/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-kube-api-access-8sr65\") pod \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") "
Jul 29 21:43:09.216974 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.216955    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-openshift-service-ca\") pod \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") "
Jul 29 21:43:09.217016 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.216985    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-maas-api-tls\") pod \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\" (UID: \"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505\") "
Jul 29 21:43:09.217389 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.217363    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06b87f01-aeb3-4cc0-b8d3-d8a8f713c505" (UID: "06b87f01-aeb3-4cc0-b8d3-d8a8f713c505"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:43:09.218866 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.218846    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-kube-api-access-8sr65" (OuterVolumeSpecName: "kube-api-access-8sr65") pod "06b87f01-aeb3-4cc0-b8d3-d8a8f713c505" (UID: "06b87f01-aeb3-4cc0-b8d3-d8a8f713c505"). InnerVolumeSpecName "kube-api-access-8sr65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:43:09.219015 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.218999    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06b87f01-aeb3-4cc0-b8d3-d8a8f713c505" (UID: "06b87f01-aeb3-4cc0-b8d3-d8a8f713c505"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:43:09.303427 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.303394    2585 generic.go:356] "Generic (PLEG): container finished" podID="06b87f01-aeb3-4cc0-b8d3-d8a8f713c505" containerID="87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905" exitCode=0
Jul 29 21:43:09.303539 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.303430    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"
Jul 29 21:43:09.303539 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.303481    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5" event={"ID":"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505","Type":"ContainerDied","Data":"87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905"}
Jul 29 21:43:09.303539 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.303530    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5" event={"ID":"06b87f01-aeb3-4cc0-b8d3-d8a8f713c505","Type":"ContainerDied","Data":"5470a5bee3a3031b1f3165bf1b4b9c8c35589ac0a9b40b11037a59337a0fff39"}
Jul 29 21:43:09.303647 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.303554    2585 scope.go:117] "RemoveContainer" containerID="87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905"
Jul 29 21:43:09.312447 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.312424    2585 scope.go:117] "RemoveContainer" containerID="87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905"
Jul 29 21:43:09.312709 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:43:09.312690    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905\": container with ID starting with 87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905 not found: ID does not exist" containerID="87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905"
Jul 29 21:43:09.312785 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.312717    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905"} err="failed to get container status \"87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905\": rpc error: code = NotFound desc = could not find container \"87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905\": container with ID starting with 87b71b207e744d4850e7cb5e4788df35cf3ebfe0f7c680fcedfdcb95c3601905 not found: ID does not exist"
Jul 29 21:43:09.317772 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.317753    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sr65\" (UniqueName: \"kubernetes.io/projected/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-kube-api-access-8sr65\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:09.317772 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.317771    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:09.317879 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.317781    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:09.324774 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.324755    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"]
Jul 29 21:43:09.327500 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.327475    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5f7d51-778dbff8dd-hbwv5"]
Jul 29 21:43:09.517242 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.517175    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:09.523469 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:09.523449    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06b87f01-aeb3-4cc0-b8d3-d8a8f713c505" path="/var/lib/kubelet/pods/06b87f01-aeb3-4cc0-b8d3-d8a8f713c505/volumes"
Jul 29 21:43:13.343244 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.343204    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8" podUID="cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" containerName="main" containerID="cri-o://fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5" gracePeriod=45
Jul 29 21:43:13.597368 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.597304    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:43:13.654021 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.653992    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-dshm\") pod \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") "
Jul 29 21:43:13.654131 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654028    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-model-cache\") pod \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") "
Jul 29 21:43:13.654131 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654099    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tls-certs\") pod \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") "
Jul 29 21:43:13.654211 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654150    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chcxj\" (UniqueName: \"kubernetes.io/projected/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-kube-api-access-chcxj\") pod \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") "
Jul 29 21:43:13.654211 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654174    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tmp-dir\") pod \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") "
Jul 29 21:43:13.654279 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654237    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-home\") pod \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\" (UID: \"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f\") "
Jul 29 21:43:13.654380 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654324    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" (UID: "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:43:13.654572 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654520    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" (UID: "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:43:13.654681 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654590    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-home" (OuterVolumeSpecName: "home") pod "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" (UID: "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:43:13.654747 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654686    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:13.654747 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654699    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:13.654747 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.654709    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:13.656086 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.656057    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-dshm" (OuterVolumeSpecName: "dshm") pod "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" (UID: "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:43:13.656203 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.656186    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" (UID: "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:43:13.656485 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.656461    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-kube-api-access-chcxj" (OuterVolumeSpecName: "kube-api-access-chcxj") pod "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" (UID: "cfbad70c-04c5-4d1d-a3e7-0a4ce574252f"). InnerVolumeSpecName "kube-api-access-chcxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:43:13.755628 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.755606    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:13.755628 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.755626    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:13.755749 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:13.755636    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chcxj\" (UniqueName: \"kubernetes.io/projected/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f-kube-api-access-chcxj\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:14.336915 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:14.336876    2585 generic.go:356] "Generic (PLEG): container finished" podID="cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" containerID="fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5" exitCode=0
Jul 29 21:43:14.337112 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:14.336928    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"
Jul 29 21:43:14.337112 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:14.336971    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8" event={"ID":"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f","Type":"ContainerDied","Data":"fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5"}
Jul 29 21:43:14.337112 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:14.337031    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8" event={"ID":"cfbad70c-04c5-4d1d-a3e7-0a4ce574252f","Type":"ContainerDied","Data":"612657ebeda2b3a6af306d884b1f1cfd00fb7cab704aebb6bdc6bb956beb6f34"}
Jul 29 21:43:14.337112 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:14.337062    2585 scope.go:117] "RemoveContainer" containerID="fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5"
Jul 29 21:43:14.347676 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:14.347515    2585 scope.go:117] "RemoveContainer" containerID="fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5"
Jul 29 21:43:14.347909 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:43:14.347807    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5\": container with ID starting with fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5 not found: ID does not exist" containerID="fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5"
Jul 29 21:43:14.347909 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:14.347834    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5"} err="failed to get container status \"fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5\": rpc error: code = NotFound desc = could not find container \"fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5\": container with ID starting with fe9b57889d086a8bd464d68e2aa78d27f4212befe105b496ab134822ff2c44e5 not found: ID does not exist"
Jul 29 21:43:14.362549 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:14.362519    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"]
Jul 29 21:43:14.370752 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:14.370728    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-eee583/rate-test-model-eee583-kserve-7f9dd87c9f-m8cz8"]
Jul 29 21:43:15.524648 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:15.524615    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" path="/var/lib/kubelet/pods/cfbad70c-04c5-4d1d-a3e7-0a4ce574252f/volumes"
Jul 29 21:43:15.694706 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:15.694674    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:15.773790 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:15.773731    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:15.972016 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:15.971987    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:16.224640 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.224576    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:16.340663 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.340634    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"]
Jul 29 21:43:16.341085 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.341068    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b87f01-aeb3-4cc0-b8d3-d8a8f713c505" containerName="maas-api"
Jul 29 21:43:16.341184 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.341089    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b87f01-aeb3-4cc0-b8d3-d8a8f713c505" containerName="maas-api"
Jul 29 21:43:16.341184 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.341104    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" containerName="main"
Jul 29 21:43:16.341184 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.341112    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" containerName="main"
Jul 29 21:43:16.341363 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.341205    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfbad70c-04c5-4d1d-a3e7-0a4ce574252f" containerName="main"
Jul 29 21:43:16.341363 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.341227    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b87f01-aeb3-4cc0-b8d3-d8a8f713c505" containerName="maas-api"
Jul 29 21:43:16.346840 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.346821    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"
Jul 29 21:43:16.349821 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.349801    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wpqrc\""
Jul 29 21:43:16.353211 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.353189    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"]
Jul 29 21:43:16.477682 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.477611    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc00ea4-a8b2-4ac7-81ce-d766060f3b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d\" (UID: \"3fc00ea4-a8b2-4ac7-81ce-d766060f3b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"
Jul 29 21:43:16.578566 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.578540    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc00ea4-a8b2-4ac7-81ce-d766060f3b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d\" (UID: \"3fc00ea4-a8b2-4ac7-81ce-d766060f3b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"
Jul 29 21:43:16.579114 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.579097    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc00ea4-a8b2-4ac7-81ce-d766060f3b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d\" (UID: \"3fc00ea4-a8b2-4ac7-81ce-d766060f3b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"
Jul 29 21:43:16.660560 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.660536    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"
Jul 29 21:43:16.795723 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:16.795700    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"]
Jul 29 21:43:16.797808 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:43:16.797769    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc00ea4_a8b2_4ac7_81ce_d766060f3b01.slice/crio-8eabc32e990d2b37e6b76fbab1db3ec04ed6286b8e2d76530202c60d5466ccda WatchSource:0}: Error finding container 8eabc32e990d2b37e6b76fbab1db3ec04ed6286b8e2d76530202c60d5466ccda: Status 404 returned error can't find the container with id 8eabc32e990d2b37e6b76fbab1db3ec04ed6286b8e2d76530202c60d5466ccda
Jul 29 21:43:17.353887 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:17.353815    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d" event={"ID":"3fc00ea4-a8b2-4ac7-81ce-d766060f3b01","Type":"ContainerStarted","Data":"ff1694bc0c8819db79b7546b8efdfdb90425fee0c2cd8ef6136ee2755ed7d81f"}
Jul 29 21:43:17.353887 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:17.353862    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d" event={"ID":"3fc00ea4-a8b2-4ac7-81ce-d766060f3b01","Type":"ContainerStarted","Data":"8eabc32e990d2b37e6b76fbab1db3ec04ed6286b8e2d76530202c60d5466ccda"}
Jul 29 21:43:17.383473 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:17.383428    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d" podStartSLOduration=1.383415798 podStartE2EDuration="1.383415798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:43:17.380593522 +0000 UTC m=+4724.424285705" watchObservedRunningTime="2026-07-29 21:43:17.383415798 +0000 UTC m=+4724.427107981"
Jul 29 21:43:18.361105 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:18.361070    2585 generic.go:356] "Generic (PLEG): container finished" podID="3fc00ea4-a8b2-4ac7-81ce-d766060f3b01" containerID="ff1694bc0c8819db79b7546b8efdfdb90425fee0c2cd8ef6136ee2755ed7d81f" exitCode=0
Jul 29 21:43:18.361520 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:18.361111    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d" event={"ID":"3fc00ea4-a8b2-4ac7-81ce-d766060f3b01","Type":"ContainerDied","Data":"ff1694bc0c8819db79b7546b8efdfdb90425fee0c2cd8ef6136ee2755ed7d81f"}
Jul 29 21:43:19.491121 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:19.491097    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"
Jul 29 21:43:19.604623 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:19.604596    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc00ea4-a8b2-4ac7-81ce-d766060f3b01-maas-api-service-ca\") pod \"3fc00ea4-a8b2-4ac7-81ce-d766060f3b01\" (UID: \"3fc00ea4-a8b2-4ac7-81ce-d766060f3b01\") "
Jul 29 21:43:19.604942 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:19.604918    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc00ea4-a8b2-4ac7-81ce-d766060f3b01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fc00ea4-a8b2-4ac7-81ce-d766060f3b01" (UID: "3fc00ea4-a8b2-4ac7-81ce-d766060f3b01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:43:19.706209 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:19.706138    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc00ea4-a8b2-4ac7-81ce-d766060f3b01-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:20.371407 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:20.371369    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"
Jul 29 21:43:20.371588 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:20.371363    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d" event={"ID":"3fc00ea4-a8b2-4ac7-81ce-d766060f3b01","Type":"ContainerDied","Data":"8eabc32e990d2b37e6b76fbab1db3ec04ed6286b8e2d76530202c60d5466ccda"}
Jul 29 21:43:20.371588 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:20.371500    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eabc32e990d2b37e6b76fbab1db3ec04ed6286b8e2d76530202c60d5466ccda"
Jul 29 21:43:26.925084 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:26.925052    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"]
Jul 29 21:43:26.925472 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:26.925260    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5" podUID="c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57" containerName="maas-api" containerID="cri-o://d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51" gracePeriod=30
Jul 29 21:43:26.953088 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:26.953062    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:27.035696 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.035655    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:27.145211 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.145173    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:27.153720 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.153694    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"]
Jul 29 21:43:27.155465 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.155441    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eee583-xpd9d"]
Jul 29 21:43:27.173874 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.173856    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:43:27.204403 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.204344    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:27.273423 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.273387    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bstfz\" (UniqueName: \"kubernetes.io/projected/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-kube-api-access-bstfz\") pod \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") "
Jul 29 21:43:27.273549 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.273532    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-maas-api-tls\") pod \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") "
Jul 29 21:43:27.273619 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.273579    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-openshift-service-ca\") pod \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\" (UID: \"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57\") "
Jul 29 21:43:27.273936 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.273906    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57" (UID: "c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:43:27.275536 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.275516    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57" (UID: "c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:43:27.275619 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.275585    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-kube-api-access-bstfz" (OuterVolumeSpecName: "kube-api-access-bstfz") pod "c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57" (UID: "c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57"). InnerVolumeSpecName "kube-api-access-bstfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:43:27.374989 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.374964    2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:27.374989 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.374987    2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:27.375119 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.374997    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bstfz\" (UniqueName: \"kubernetes.io/projected/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57-kube-api-access-bstfz\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\""
Jul 29 21:43:27.404275 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.404242    2585 generic.go:356] "Generic (PLEG): container finished" podID="c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57" containerID="d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51" exitCode=0
Jul 29 21:43:27.404406 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.404284    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5" event={"ID":"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57","Type":"ContainerDied","Data":"d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51"}
Jul 29 21:43:27.404406 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.404371    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5" event={"ID":"c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57","Type":"ContainerDied","Data":"676ecdcd50f126f3812321822f031164f59e479dfe71ab536ae1f15320c5fbbb"}
Jul 29 21:43:27.404406 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.404294    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"
Jul 29 21:43:27.404522 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.404400    2585 scope.go:117] "RemoveContainer" containerID="d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51"
Jul 29 21:43:27.413284 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.413268    2585 scope.go:117] "RemoveContainer" containerID="d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51"
Jul 29 21:43:27.413592 ip-10-0-136-123 kubenswrapper[2585]: E0729 21:43:27.413566    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51\": container with ID starting with d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51 not found: ID does not exist" containerID="d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51"
Jul 29 21:43:27.413661 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.413601    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51"} err="failed to get container status \"d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51\": rpc error: code = NotFound desc = could not find container \"d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51\": container with ID starting with d23b85cd0d5b98bf803bcd00243f7281cb4a9da5d7d2dc27958c400438556f51 not found: ID does not exist"
Jul 29 21:43:27.426610 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.426584    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"]
Jul 29 21:43:27.432601 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.432578    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eee583-56697c7bfb-g6dd5"]
Jul 29 21:43:27.524088 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.524065    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fc00ea4-a8b2-4ac7-81ce-d766060f3b01" path="/var/lib/kubelet/pods/3fc00ea4-a8b2-4ac7-81ce-d766060f3b01/volumes"
Jul 29 21:43:27.524512 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.524490    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57" path="/var/lib/kubelet/pods/c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57/volumes"
Jul 29 21:43:27.543471 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:27.543450    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:33.650136 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:33.650100    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:33.744665 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:33.744630    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:33.980496 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:33.980465    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:34.178811 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:34.178783    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:34.611994 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:34.611969    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84d9f98d8b-8kch5_d4280eee-f663-46a7-8ba7-f0f97d53160d/authorino/0.log"
Jul 29 21:43:40.441817 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:40.441783    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5458c6456b-hbrtq_9659f107-7c42-47d6-b32e-8b5ca15a2335/manager/0.log"
Jul 29 21:43:40.940447 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:40.940417    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74f7855d79-sn5d6_39b62403-cc9e-4d4e-bdd3-ee0479a5509f/copy-manifests/0.log"
Jul 29 21:43:40.959291 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:40.959261    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74f7855d79-sn5d6_39b62403-cc9e-4d4e-bdd3-ee0479a5509f/manager/0.log"
Jul 29 21:43:41.462672 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:41.462645    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74b678b674-2wcwv_12064095-bb7a-4e22-93b8-a22c90eacd4f/maas-api/0.log"
Jul 29 21:43:43.310284 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:43.310254    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84d9f98d8b-8kch5_d4280eee-f663-46a7-8ba7-f0f97d53160d/authorino/0.log"
Jul 29 21:43:44.159342 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:44.159303    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5g9ct_2eab4c72-f8a5-45e7-88e6-e92919329e31/manager/0.log"
Jul 29 21:43:44.768810 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:44.768778    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b588c995f-5f7xf_8a5ce228-0a71-40de-aae7-03f29ce44897/kube-auth-proxy/0.log"
Jul 29 21:43:45.746208 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:45.746176    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-c586s_df5bc3cf-a925-49e8-b0ad-148c62e728d3/main/0.log"
Jul 29 21:43:46.216782 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:46.216738    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4fgx6_f992bd32-9946-40df-9424-4f9b1cd174ec/main/0.log"
Jul 29 21:43:46.339154 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:46.339126    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-q5cjm_a6d87ff8-aa15-4e2a-aac8-a508ff9eb0ea/main/0.log"
Jul 29 21:43:51.896991 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:51.896962    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:51.977133 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:51.977103    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log" limit=1024
Jul 29 21:43:54.733350 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:54.733294    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9xh8g_b0c1248c-5bb1-4cd3-bade-adc29339979c/global-pull-secret-syncer/0.log"
Jul 29 21:43:54.886274 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:54.886240    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h9t4t_26eecc01-788c-477e-b50a-5553413d3dd8/konnectivity-agent/0.log"
Jul 29 21:43:55.008391 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:43:55.008293    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-123.ec2.internal_1f03ef72ad08cd4666542db4e9e3bb71/haproxy/0.log"
Jul 29 21:44:00.387129 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:00.387094    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84d9f98d8b-8kch5_d4280eee-f663-46a7-8ba7-f0f97d53160d/authorino/0.log"
Jul 29 21:44:00.819736 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:00.819693    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5g9ct_2eab4c72-f8a5-45e7-88e6-e92919329e31/manager/0.log"
Jul 29 21:44:02.590840 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.590808    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4b6e493c-ec2d-4a36-8815-75abe62767f7/alertmanager/0.log"
Jul 29 21:44:02.620386 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.620355    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4b6e493c-ec2d-4a36-8815-75abe62767f7/config-reloader/0.log"
Jul 29 21:44:02.641823 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.641805    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4b6e493c-ec2d-4a36-8815-75abe62767f7/kube-rbac-proxy-web/0.log"
Jul 29 21:44:02.662550 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.662531    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4b6e493c-ec2d-4a36-8815-75abe62767f7/kube-rbac-proxy/0.log"
Jul 29 21:44:02.682821 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.682796    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4b6e493c-ec2d-4a36-8815-75abe62767f7/kube-rbac-proxy-metric/0.log"
Jul 29 21:44:02.709633 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.709578    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4b6e493c-ec2d-4a36-8815-75abe62767f7/prom-label-proxy/0.log"
Jul 29 21:44:02.730752 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.730733    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4b6e493c-ec2d-4a36-8815-75abe62767f7/init-config-reloader/0.log"
Jul 29 21:44:02.843034 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.843014    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-vzwdj_3bdfb728-4abe-469f-9c5b-9966b0f8f429/kube-state-metrics/0.log"
Jul 29 21:44:02.858767 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.858751    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-vzwdj_3bdfb728-4abe-469f-9c5b-9966b0f8f429/kube-rbac-proxy-main/0.log"
Jul 29 21:44:02.880819 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.880781    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-vzwdj_3bdfb728-4abe-469f-9c5b-9966b0f8f429/kube-rbac-proxy-self/0.log"
Jul 29 21:44:02.970253 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.970230    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2zzj_f555377d-dc85-4e77-8da4-a2a4162dc45e/node-exporter/0.log"
Jul 29 21:44:02.995070 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:02.995046    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2zzj_f555377d-dc85-4e77-8da4-a2a4162dc45e/kube-rbac-proxy/0.log"
Jul 29 21:44:03.024492 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.024474    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2zzj_f555377d-dc85-4e77-8da4-a2a4162dc45e/init-textfile/0.log"
Jul 29 21:44:03.330772 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.330697    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26b8dfa8-6577-40d5-8ad5-197f5d97660d/prometheus/0.log"
Jul 29 21:44:03.351379 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.351356    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26b8dfa8-6577-40d5-8ad5-197f5d97660d/config-reloader/0.log"
Jul 29 21:44:03.372721 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.372693    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26b8dfa8-6577-40d5-8ad5-197f5d97660d/thanos-sidecar/0.log"
Jul 29 21:44:03.392550 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.392530    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26b8dfa8-6577-40d5-8ad5-197f5d97660d/kube-rbac-proxy-web/0.log"
Jul 29 21:44:03.412188 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.412168    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26b8dfa8-6577-40d5-8ad5-197f5d97660d/kube-rbac-proxy/0.log"
Jul 29 21:44:03.442770 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.442747    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26b8dfa8-6577-40d5-8ad5-197f5d97660d/kube-rbac-proxy-thanos/0.log"
Jul 29 21:44:03.464538 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.464517    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26b8dfa8-6577-40d5-8ad5-197f5d97660d/init-config-reloader/0.log"
Jul 29 21:44:03.533367 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.533340    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-cpx9c_40b60c36-dfff-4809-802e-dd224bde115d/prometheus-operator-admission-webhook/0.log"
Jul 29 21:44:03.638594 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.638523    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf4864fc6-z597l_a30a62ac-86d3-42e6-8cdf-a932c07ebdc0/thanos-query/0.log"
Jul 29 21:44:03.659604 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.659585    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf4864fc6-z597l_a30a62ac-86d3-42e6-8cdf-a932c07ebdc0/kube-rbac-proxy-web/0.log"
Jul 29 21:44:03.681253 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.681228    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf4864fc6-z597l_a30a62ac-86d3-42e6-8cdf-a932c07ebdc0/kube-rbac-proxy/0.log"
Jul 29 21:44:03.704398 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.704376    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf4864fc6-z597l_a30a62ac-86d3-42e6-8cdf-a932c07ebdc0/prom-label-proxy/0.log"
Jul 29 21:44:03.724223 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.724205    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf4864fc6-z597l_a30a62ac-86d3-42e6-8cdf-a932c07ebdc0/kube-rbac-proxy-rules/0.log"
Jul 29 21:44:03.743324 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:03.743304    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf4864fc6-z597l_a30a62ac-86d3-42e6-8cdf-a932c07ebdc0/kube-rbac-proxy-metrics/0.log"
Jul 29 21:44:05.023719 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.023687    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"]
Jul 29 21:44:05.024376 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.024194    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57" containerName="maas-api"
Jul 29 21:44:05.024376 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.024214    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57" containerName="maas-api"
Jul 29 21:44:05.024376 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.024261    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc00ea4-a8b2-4ac7-81ce-d766060f3b01" containerName="revoke-keys"
Jul 29 21:44:05.024376 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.024270    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc00ea4-a8b2-4ac7-81ce-d766060f3b01" containerName="revoke-keys"
Jul 29 21:44:05.024662 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.024384    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6db15fd-e9b8-4c5b-a6e2-4a658ccaeb57" containerName="maas-api"
Jul 29 21:44:05.024662 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.024397    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc00ea4-a8b2-4ac7-81ce-d766060f3b01" containerName="revoke-keys"
Jul 29 21:44:05.027437 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.027418    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.030013 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.029985    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pwwzw\"/\"default-dockercfg-5976b\""
Jul 29 21:44:05.030128 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.030028    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pwwzw\"/\"openshift-service-ca.crt\""
Jul 29 21:44:05.031233 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.031210    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pwwzw\"/\"kube-root-ca.crt\""
Jul 29 21:44:05.036236 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.036212    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"]
Jul 29 21:44:05.070840 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.070814    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-lib-modules\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.070933 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.070847    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-sys\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.070976 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.070937    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-proc\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.071012 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.070996    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-podres\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.071051 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.071023    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sr4x\" (UniqueName: \"kubernetes.io/projected/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-kube-api-access-8sr4x\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.171796 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.171773    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-proc\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.171916 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.171817    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-podres\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.171916 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.171844    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sr4x\" (UniqueName: \"kubernetes.io/projected/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-kube-api-access-8sr4x\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.171916 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.171887    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-lib-modules\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.171916 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.171891    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-proc\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.172108 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.171921    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-sys\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.172108 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.171945    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-podres\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.172108 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.172011    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-lib-modules\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.172108 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.172010    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-sys\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.180503 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.180484    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sr4x\" (UniqueName: \"kubernetes.io/projected/0f5df3a6-7454-4b8c-9167-c7843f2b5b80-kube-api-access-8sr4x\") pod \"perf-node-gather-daemonset-5xspp\" (UID: \"0f5df3a6-7454-4b8c-9167-c7843f2b5b80\") " pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.338921 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.338873    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.463897 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.463873    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"]
Jul 29 21:44:05.466425 ip-10-0-136-123 kubenswrapper[2585]: W0729 21:44:05.466399    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f5df3a6_7454_4b8c_9167_c7843f2b5b80.slice/crio-f98b4298abbc4da17618117cf6bce7ec17d080f95aaeb386a849cf4adf964466 WatchSource:0}: Error finding container f98b4298abbc4da17618117cf6bce7ec17d080f95aaeb386a849cf4adf964466: Status 404 returned error can't find the container with id f98b4298abbc4da17618117cf6bce7ec17d080f95aaeb386a849cf4adf964466
Jul 29 21:44:05.594968 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.594908    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp" event={"ID":"0f5df3a6-7454-4b8c-9167-c7843f2b5b80","Type":"ContainerStarted","Data":"7ab77019bf57feb68b8c33253e1ef9166d0e4de96f78555321480958530380d9"}
Jul 29 21:44:05.594968 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.594965    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp" event={"ID":"0f5df3a6-7454-4b8c-9167-c7843f2b5b80","Type":"ContainerStarted","Data":"f98b4298abbc4da17618117cf6bce7ec17d080f95aaeb386a849cf4adf964466"}
Jul 29 21:44:05.595100 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.594997    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:05.614429 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.614389    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp" podStartSLOduration=0.614376633 podStartE2EDuration="614.376633ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:44:05.612031731 +0000 UTC m=+4772.655723928" watchObservedRunningTime="2026-07-29 21:44:05.614376633 +0000 UTC m=+4772.658068816"
Jul 29 21:44:05.725931 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.725905    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/2.log"
Jul 29 21:44:05.733922 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:05.733899    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-7lq85_0248d8c6-87bb-4df2-8f61-f80a2c51469b/console-operator/3.log"
Jul 29 21:44:07.964985 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:07.964947    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cgmsd_c106d721-0381-4f51-ad74-dcb1804221fc/dns/0.log"
Jul 29 21:44:07.987527 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:07.987503    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cgmsd_c106d721-0381-4f51-ad74-dcb1804221fc/kube-rbac-proxy/0.log"
Jul 29 21:44:08.100579 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:08.100557    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qm8fd_9191e46f-8231-4d1c-8104-c679ad0e2eeb/dns-node-resolver/0.log"
Jul 29 21:44:08.688122 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:08.688091    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k58wb_7cf47463-470a-4c86-8301-a838734a4963/node-ca/0.log"
Jul 29 21:44:09.938618 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:09.938576    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b588c995f-5f7xf_8a5ce228-0a71-40de-aae7-03f29ce44897/kube-auth-proxy/0.log"
Jul 29 21:44:10.997471 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:10.997436    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-knr4r_48bb1c6c-0d0e-436c-af8d-747647703600/serve-healthcheck-canary/0.log"
Jul 29 21:44:11.568283 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:11.568255    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggdgh_b8f1b3cb-a4e4-401f-aeb2-34451e025fcf/kube-rbac-proxy/0.log"
Jul 29 21:44:11.587047 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:11.587026    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggdgh_b8f1b3cb-a4e4-401f-aeb2-34451e025fcf/exporter/0.log"
Jul 29 21:44:11.609255 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:11.609237    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pwwzw/perf-node-gather-daemonset-5xspp"
Jul 29 21:44:11.610714 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:11.610691    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggdgh_b8f1b3cb-a4e4-401f-aeb2-34451e025fcf/extractor/0.log"
Jul 29 21:44:14.445646 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:14.445612    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5458c6456b-hbrtq_9659f107-7c42-47d6-b32e-8b5ca15a2335/manager/0.log"
Jul 29 21:44:14.779515 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:14.779489    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74f7855d79-sn5d6_39b62403-cc9e-4d4e-bdd3-ee0479a5509f/manager/0.log"
Jul 29 21:44:14.814441 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:14.814418    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74f7855d79-sn5d6_39b62403-cc9e-4d4e-bdd3-ee0479a5509f/copy-manifests/0.log"
Jul 29 21:44:15.520718 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:15.520693    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b588c995f-5f7xf_8a5ce228-0a71-40de-aae7-03f29ce44897/kube-auth-proxy/0.log"
Jul 29 21:44:16.312008 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:16.311982    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c7786b84c-pz88f_59be7ae2-f437-47c5-9d2f-83d34eb48321/manager/0.log"
Jul 29 21:44:17.469497 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:17.469464    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b588c995f-5f7xf_8a5ce228-0a71-40de-aae7-03f29ce44897/kube-auth-proxy/0.log"
Jul 29 21:44:21.630499 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:21.630466    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl_e7bc6de2-919c-4f33-98ea-1a838956bd8f/kube-storage-version-migrator-operator/1.log"
Jul 29 21:44:21.631776 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:21.631751    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-t2dcl_e7bc6de2-919c-4f33-98ea-1a838956bd8f/kube-storage-version-migrator-operator/0.log"
Jul 29 21:44:22.613902 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:22.613865    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6hgg4_1f01f25d-0021-40f3-a536-01a921d1557d/kube-multus/0.log"
Jul 29 21:44:22.637690 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:22.637664    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7z62g_a78b2c7b-217a-4490-921d-84357434b580/kube-multus-additional-cni-plugins/0.log"
Jul 29 21:44:22.662978 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:22.662944    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7z62g_a78b2c7b-217a-4490-921d-84357434b580/egress-router-binary-copy/0.log"
Jul 29 21:44:22.699503 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:22.699406    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7z62g_a78b2c7b-217a-4490-921d-84357434b580/cni-plugins/0.log"
Jul 29 21:44:22.728818 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:22.728797    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7z62g_a78b2c7b-217a-4490-921d-84357434b580/bond-cni-plugin/0.log"
Jul 29 21:44:22.749549 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:22.749532    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7z62g_a78b2c7b-217a-4490-921d-84357434b580/routeoverride-cni/0.log"
Jul 29 21:44:22.770222 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:22.770204    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7z62g_a78b2c7b-217a-4490-921d-84357434b580/whereabouts-cni-bincopy/0.log"
Jul 29 21:44:22.790178 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:22.790158    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7z62g_a78b2c7b-217a-4490-921d-84357434b580/whereabouts-cni/0.log"
Jul 29 21:44:23.253908 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:23.253878    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7sqwj_94bf5594-165a-4efb-ba8c-a628ea34eabf/network-metrics-daemon/0.log"
Jul 29 21:44:23.278885 ip-10-0-136-123 kubenswrapper[2585]: I0729 21:44:23.278858    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7sqwj_94bf5594-165a-4efb-ba8c-a628ea34eabf/kube-rbac-proxy/0.log"