Jul 28 22:09:57.764380 ip-10-0-129-50 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 22:09:58.136392 ip-10-0-129-50 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 22:09:58.136392 ip-10-0-129-50 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 22:09:58.136392 ip-10-0-129-50 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 22:09:58.136392 ip-10-0-129-50 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 22:09:58.136392 ip-10-0-129-50 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 22:09:58.138109 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.137954    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 22:09:58.143058 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143042    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:09:58.143058 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143057    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143061    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143067    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143071    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143075    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143078    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143080    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143083    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143086    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143089    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143091    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143094    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143108    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143111    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143114    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143117    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143128    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143132    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143135    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143138    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:09:58.143154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143141    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143143    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143146    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143149    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143152    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143154    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143157    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143160    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143163    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143166    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143168    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143171    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143174    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143177    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143179    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143182    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143185    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143187    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143190    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:09:58.143631 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143192    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143195    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143199    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143203    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143206    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143209    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143212    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143214    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143217    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143219    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143222    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143225    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143228    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143230    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143233    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143236    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143239    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143242    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143245    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:09:58.144131 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143248    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143251    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143253    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143256    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143258    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143261    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143264    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143266    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143269    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143272    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143274    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143276    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143280    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143282    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143285    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143287    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143290    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143293    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143295    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143298    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:09:58.144596 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143301    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143303    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143306    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143308    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143311    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143315    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143319    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143321    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143326    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143330    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143333    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143336    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143339    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143341    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143344    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143346    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143349    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143351    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143354    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.143356    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:09:58.145073 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144061    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144068    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144070    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144073    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144076    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144079    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144082    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144085    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144087    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144090    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144092    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144095    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144114    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144117    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144120    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144122    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144125    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144128    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144130    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144133    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:09:58.145570 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144137    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144140    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144144    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144146    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144149    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144152    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144155    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144157    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144160    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144162    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144165    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144167    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144170    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144172    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144175    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144177    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144181    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144184    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144186    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:09:58.146068 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144189    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144192    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144196    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144199    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144201    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144205    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144210    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144213    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144216    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144219    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144221    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144224    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144226    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144230    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144233    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144235    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144238    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144241    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144243    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144246    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:09:58.146547 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144249    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144251    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144254    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144256    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144259    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144262    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144264    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144267    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144270    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144272    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144275    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144279    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144285    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144288    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144291    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144293    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144296    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144298    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144301    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:09:58.147124 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144303    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144306    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144308    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144317    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144320    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144322    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144326    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144330    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144332    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144334    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144337    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144340    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144343    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144345    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144348    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144350    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144353    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144356    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144358    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144361    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:09:58.147600 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144363    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144765    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144771    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144774    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144778    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144781    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144784    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144787    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144790    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144793    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144796    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144798    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144801    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144804    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144807    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144811    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144815    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144818    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144821    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:09:58.148088 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144823    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144826    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144829    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144832    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144835    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144837    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144840    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144843    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144845    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144848    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144850    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144853    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144856    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144858    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144861    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144863    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144865    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144868    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144871    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144874    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:09:58.148560 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144876    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144879    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144882    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144884    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144887    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144889    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144892    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144894    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144897    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144900    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144903    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144905    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144908    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144910    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144913    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144915    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144918    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144920    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144922    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144925    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:09:58.149034 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144927    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144929    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144933    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144936    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144939    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144942    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144944    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144947    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144950    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144953    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144956    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144958    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144961    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144964    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144967    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144969    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144972    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144975    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144979    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144981    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:09:58.149538 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144985    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144990    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144993    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144996    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.144998    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145001    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145003    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145006    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145008    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145011    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145013    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145016    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145018    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145020    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145023    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145025    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145029    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145033    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145035    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:09:58.150012 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145038    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145040    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145128    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145136    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145142    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145146    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145151    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145154    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145159    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145163    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145166    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145169    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145175    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145179    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145183    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145186    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145189    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145193    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145196    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145199    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145203    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145206    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145210    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145213    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 22:09:58.150478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145216    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145220    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145223    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145226    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145230    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145233    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145236    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145240    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145243    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145246    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145250    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145253    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145256    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145259    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145262    2560 flags.go:64] FLAG: --enable-server="true"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145265    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145269    2560 flags.go:64] FLAG: --event-burst="100"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145273    2560 flags.go:64] FLAG: --event-qps="50"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145276    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145278    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145283    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145286    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145290    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145294    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145297    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 28 22:09:58.151077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145299    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145302    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145306    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145309    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145311    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145314    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145317    2560 flags.go:64] FLAG: --feature-gates=""
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145321    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145324    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145327    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145330    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145333    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145336    2560 flags.go:64] FLAG: --help="false"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145339    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145342    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145345    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145348    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145351    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145354    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145357    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145360    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145363    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145366    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 22:09:58.151732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145369    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145372    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145375    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145378    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145382    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145385    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145389    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145392    2560 flags.go:64] FLAG: --lock-file=""
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145395    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145398    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145401    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145411    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145414    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145417    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145420    2560 flags.go:64] FLAG: --logging-format="text"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145423    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145426    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145429    2560 flags.go:64] FLAG: --manifest-url=""
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145432    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145436    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145439    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145443    2560 flags.go:64] FLAG: --max-pods="110"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145446    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145449    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145452    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 22:09:58.152323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145455    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145458    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145461    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145464    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145471    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145474    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145477    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145480    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145484    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145489    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145492    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145495    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145499    2560 flags.go:64] FLAG: --port="10250"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145504    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145507    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03cd2a77f1060ff97"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145510    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145513    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145516    2560 flags.go:64] FLAG: --register-node="true"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145519    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145522    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145526    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145528    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145531    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145535    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145538    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 22:09:58.152925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145541    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145544    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145547    2560 flags.go:64] FLAG: --runonce="false"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145550    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145553    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145555    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145558    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145561    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145564    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145567    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145570    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145573    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145576    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145579    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145582    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145585    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145589    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145592    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145597    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145601    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145605    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145610    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145613    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145616    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145619    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 22:09:58.153557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145622    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145625    2560 flags.go:64] FLAG: --v="2"
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145629    2560 flags.go:64] FLAG: --version="false"
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145633    2560 flags.go:64] FLAG: --vmodule=""
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145636    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.145639    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145755    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145758    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145762    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145765    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145768    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145771    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145774    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145777    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145780    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145782    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145785    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145787    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145790    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145793    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145797    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:09:58.154542 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145801    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145804    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145807    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145811    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145814    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145820    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145825    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145828    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145831    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145834    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145836    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145839    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145842    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145844    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145847    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145850    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145852    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145855    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145857    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:09:58.155248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145860    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145862    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145865    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145868    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145870    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145873    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145875    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145878    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145881    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145883    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145886    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145889    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145891    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145894    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145896    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145899    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145902    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145905    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145908    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145912    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:09:58.155723 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145915    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145918    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145921    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145924    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145926    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145929    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145931    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145934    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145936    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145939    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145942    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145944    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145947    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145949    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145952    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145955    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145957    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145959    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145962    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145965    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:09:58.156232 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145967    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145970    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145973    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145975    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145978    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145980    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145983    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145985    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145988    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145990    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145995    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.145999    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146002    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146004    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146007    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146010    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146012    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146015    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146017    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:09:58.156742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146020    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146022    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146025    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146029    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146032    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.146035    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.146588    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.154805    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.154828    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.155808    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.155824    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155892    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155898    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155902    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:09:58.157221 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155905    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155908    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155912    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155917    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155920    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155923    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155925    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155928    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155931    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155933    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155936    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155939    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155943    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155947    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155950    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155954    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155957    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155960    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155962    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:09:58.157566 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155965    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155969    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155971    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155974    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155977    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155980    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155983    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155987    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155990    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155994    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.155997    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156000    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156002    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156005    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156008    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156010    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156013    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156016    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156018    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156021    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:09:58.158009 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156024    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156026    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156029    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156031    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156034    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156037    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156039    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156042    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156044    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156048    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156050    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156053    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156056    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156058    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156061    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156064    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156066    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156069    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156071    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156074    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:09:58.158504 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156077    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156080    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156083    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156085    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156088    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156092    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156108    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156113    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156116    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156119    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156122    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156125    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156128    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156130    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156133    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156136    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156139    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156142    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156144    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156147    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:09:58.158999 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156150    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156153    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156155    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156158    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156160    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156163    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156166    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156168    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156170    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156173    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156175    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156179    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156181    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156184    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156187    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156189    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:09:58.159524 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156192    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.156198    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156350    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156355    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156358    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156361    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156364    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156367    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156369    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156373    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156377    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156379    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156382    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156385    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:09:58.159940 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156389    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156391    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156394    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156397    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156399    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156402    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156404    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156407    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156410    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156412    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156415    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156418    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156420    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156423    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156427    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156429    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156432    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156434    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156437    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156439    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:09:58.160314 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156442    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156444    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156447    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156450    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156453    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156455    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156457    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156460    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156463    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156465    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156468    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156471    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156473    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156476    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156478    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156481    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156483    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156486    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156488    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156491    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:09:58.160795 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156493    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156496    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156498    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156501    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156503    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156506    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156509    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156512    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156515    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156517    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156519    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156522    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156524    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156527    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156529    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156532    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156535    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156537    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156540    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156542    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:09:58.161394 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156545    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156547    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156550    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156552    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156555    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156557    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156560    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156563    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156566    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156568    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156571    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156573    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156575    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156578    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156580    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156583    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156585    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156588    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156591    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:09:58.161875 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156595    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156598    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156601    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156604    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156607    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156611    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156614    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:58.156617    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.156622    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.156739    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.158886    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.160141    2560 server.go:1019] "Starting client certificate rotation"
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.160235    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 22:09:58.162353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.160268    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 22:09:58.162705 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.162412    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 22:09:58.166699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.166685    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 22:09:58.179193 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.179170    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 28 22:09:58.181222 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.181209    2560 log.go:25] "Validated CRI v1 image API"
Jul 28 22:09:58.181913 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.181898    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 22:09:58.184339 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.184320    2560 fs.go:135] Filesystem UUIDs: map[50e280c0-295f-49e7-9aa2-6d0ea6cd15a5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e3d1db18-d5b8-4465-b921-854ef2c42dc9:/dev/nvme0n1p4]
Jul 28 22:09:58.184406 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.184339    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 22:09:58.186248 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.186232    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 22:09:58.190612 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.190496    2560 manager.go:216] Machine: {Timestamp:2026-07-28 22:09:58.189186032 +0000 UTC m=+0.311533938 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099131 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2949980b6d361d87bcf546258cb5c7 SystemUUID:ec294998-0b6d-361d-87bc-f546258cb5c7 BootID:2ad940fe-5251-4686-8d09-346516a3863a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:80:42:af:31:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:80:42:af:31:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:82:7a:f0:f8:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 22:09:58.190612 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.190610    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 22:09:58.190711 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.190690    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 22:09:58.193333 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.193306    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 22:09:58.193463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.193335    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-50.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 22:09:58.194036 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.194026    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 22:09:58.194064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.194037    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 22:09:58.194064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.194046    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 22:09:58.194725 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.194716    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 22:09:58.195687 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.195677    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 22:09:58.198161 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.198146    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 22:09:58.198283 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.198273    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 22:09:58.198324 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.198304    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 22:09:58.198324 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.198314    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 22:09:58.198385 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.198329    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 22:09:58.198385 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.198338    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 28 22:09:58.198385 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.198347    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 22:09:58.198770 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.198757    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 22:09:58.198812 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.198784    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 22:09:58.200873 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.200855    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 22:09:58.202725 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.202711    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 22:09:58.202793 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.202741    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 22:09:58.204411 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204398    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 22:09:58.204456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204415    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 22:09:58.204456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204422    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 22:09:58.204456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204427    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 22:09:58.204456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204438    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 22:09:58.204456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204446    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 22:09:58.204592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204458    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 22:09:58.204592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204466    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 22:09:58.204592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204473    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 22:09:58.204592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204479    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 22:09:58.204592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204488    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 22:09:58.204592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.204493    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 22:09:58.205227 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.205217    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 22:09:58.205265 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.205229    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 22:09:58.206901 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.206879    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z7zbk"
Jul 28 22:09:58.208668 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.208655    2560 server.go:1268] "Started kubelet"
Jul 28 22:09:58.208928 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.208905    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-50.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 22:09:58.208928 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.208911    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 22:09:58.209112 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.209050    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 22:09:58.209180 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.209136    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 22:09:58.210073 ip-10-0-129-50 systemd[1]: Started Kubernetes Kubelet.
Jul 28 22:09:58.210277 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.210094    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 22:09:58.210277 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.210113    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 22:09:58.211451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.211436    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 22:09:58.211993 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.211978    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 22:09:58.214735 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.214717    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z7zbk"
Jul 28 22:09:58.215165 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.215146    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 22:09:58.215652 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.214691    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-50.ec2.internal.18c6939b760ac78e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-50.ec2.internal,UID:ip-10-0-129-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-50.ec2.internal,},FirstTimestamp:2026-07-28 22:09:58.208612238 +0000 UTC m=+0.330960122,LastTimestamp:2026-07-28 22:09:58.208612238 +0000 UTC m=+0.330960122,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-50.ec2.internal,}"
Jul 28 22:09:58.219659 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.219645    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 22:09:58.219717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.219681    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 22:09:58.219774 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.219753    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:58.220162 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220148    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 22:09:58.220162 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220149    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 22:09:58.220295 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220177    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 22:09:58.220354 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220309    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 22:09:58.220354 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220318    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 22:09:58.220630 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220601    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 22:09:58.220630 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220620    2560 factory.go:55] Registering systemd factory
Jul 28 22:09:58.220630 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220629    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 28 22:09:58.220854 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220827    2560 factory.go:153] Registering CRI-O factory
Jul 28 22:09:58.220854 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220849    2560 factory.go:223] Registration of the crio container factory successfully
Jul 28 22:09:58.220938 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220872    2560 factory.go:103] Registering Raw factory
Jul 28 22:09:58.220938 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.220888    2560 manager.go:1196] Started watching for new ooms in manager
Jul 28 22:09:58.221341 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.221325    2560 manager.go:349] Starting recovery of all containers
Jul 28 22:09:58.228774 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.228741    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:09:58.232056 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.232035    2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-50.ec2.internal\" not found" node="ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.233324 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.233310    2560 manager.go:354] Recovery completed
Jul 28 22:09:58.237738 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.237726    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:09:58.239501 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.239487    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:09:58.239568 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.239513    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:09:58.239568 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.239523    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:09:58.240009 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.239995    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 22:09:58.240009 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.240009    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 22:09:58.240091 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.239999    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:09:58.240091 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.240026    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 22:09:58.240714 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.240702    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:09:58.240782 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.240724    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:09:58.240782 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.240734    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:09:58.241717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.241703    2560 policy_none.go:49] "None policy: Start"
Jul 28 22:09:58.241717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.241720    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 22:09:58.241817 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.241730    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 22:09:58.242991 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.242981    2560 policy_none.go:47] "Start"
Jul 28 22:09:58.279872 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.279695    2560 manager.go:337] "Starting Device Plugin manager"
Jul 28 22:09:58.279872 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.279851    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 22:09:58.279872 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.279864    2560 server.go:87] "Starting device plugin registration server"
Jul 28 22:09:58.280091 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.280079    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 22:09:58.280138 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.280091    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 22:09:58.280217 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.280205    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 22:09:58.280315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.280297    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 22:09:58.280315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.280311    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 22:09:58.280784 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.280764    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 22:09:58.280888 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.280810    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:58.385084 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.385050    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:09:58.386484 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.386167    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:09:58.386484 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.386199    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:09:58.386484 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.386209    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:09:58.386484 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.386238    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.392555 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.392539    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.392606 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.392563    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-50.ec2.internal\": node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:58.399132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.399108    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 22:09:58.400428 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.400406    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 22:09:58.400525 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.400455    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 22:09:58.400525 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.400481    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 22:09:58.400608 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.400539    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 22:09:58.402333 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.402314    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:09:58.407212 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.407192    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:58.503521 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.503470    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"]
Jul 28 22:09:58.503685 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.503584    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:09:58.505224 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.505206    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:09:58.505300 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.505238    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:09:58.505300 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.505249    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:09:58.506596 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.506580    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:09:58.506738 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.506722    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.506804 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.506760    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:09:58.507265 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.507239    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:58.507376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.507356    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:09:58.507446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.507389    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:09:58.507446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.507406    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:09:58.507446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.507411    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:09:58.507446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.507433    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:09:58.507446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.507443    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:09:58.508639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.508623    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.508726 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.508654    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:09:58.509306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.509288    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:09:58.509410 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.509314    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:09:58.509410 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.509329    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:09:58.522037 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.522018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ace7f39d4fac06f2546ed5091afdf977-config\") pod \"kube-apiserver-proxy-ip-10-0-129-50.ec2.internal\" (UID: \"ace7f39d4fac06f2546ed5091afdf977\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.522133 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.522049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ada9d69552da923316c419a39e053e4a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"ada9d69552da923316c419a39e053e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.522133 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.522079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ada9d69552da923316c419a39e053e4a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"ada9d69552da923316c419a39e053e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.522733 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.522716    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-50.ec2.internal\" not found" node="ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.526840 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.526824    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-50.ec2.internal\" not found" node="ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.607376 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.607353    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:58.622756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.622732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ada9d69552da923316c419a39e053e4a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"ada9d69552da923316c419a39e053e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.622845 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.622775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ada9d69552da923316c419a39e053e4a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"ada9d69552da923316c419a39e053e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.622845 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.622794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ace7f39d4fac06f2546ed5091afdf977-config\") pod \"kube-apiserver-proxy-ip-10-0-129-50.ec2.internal\" (UID: \"ace7f39d4fac06f2546ed5091afdf977\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.622845 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.622743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ada9d69552da923316c419a39e053e4a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"ada9d69552da923316c419a39e053e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.622967 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.622858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ace7f39d4fac06f2546ed5091afdf977-config\") pod \"kube-apiserver-proxy-ip-10-0-129-50.ec2.internal\" (UID: \"ace7f39d4fac06f2546ed5091afdf977\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.622967 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.622884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ada9d69552da923316c419a39e053e4a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"ada9d69552da923316c419a39e053e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.708487 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.708406    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:58.809191 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.809155    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:58.825840 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.825818    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.830754 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:58.830583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:58.912720 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:58.912670    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:59.014132 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:59.014030    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:59.115200 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:59.115168    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:59.159357 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.159323    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 22:09:59.159808 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.159511    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 22:09:59.159808 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.159516    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 22:09:59.215696 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:09:59.215665    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Jul 28 22:09:59.219604 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.219577    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 22:04:58 +0000 UTC" deadline="2028-05-14 21:34:09.758874908 +0000 UTC"
Jul 28 22:09:59.219679 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.219605    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15743h24m10.539273593s"
Jul 28 22:09:59.225577 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.225562    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:09:59.320253 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.320233    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:59.331655 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.331629    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 22:09:59.333374 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.333355    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Jul 28 22:09:59.342415 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.342396    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 22:09:59.374241 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:59.374204    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada9d69552da923316c419a39e053e4a.slice/crio-5a1958ba3b55018fc53f8b5bcea157f002464f2519ca12de261d0b3d057f38ac WatchSource:0}: Error finding container 5a1958ba3b55018fc53f8b5bcea157f002464f2519ca12de261d0b3d057f38ac: Status 404 returned error can't find the container with id 5a1958ba3b55018fc53f8b5bcea157f002464f2519ca12de261d0b3d057f38ac
Jul 28 22:09:59.375268 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:09:59.375243    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace7f39d4fac06f2546ed5091afdf977.slice/crio-c5f5789516f27d7ce7a913dc55b77ab6c9a1127b609abd89916b93c2216531f1 WatchSource:0}: Error finding container c5f5789516f27d7ce7a913dc55b77ab6c9a1127b609abd89916b93c2216531f1: Status 404 returned error can't find the container with id c5f5789516f27d7ce7a913dc55b77ab6c9a1127b609abd89916b93c2216531f1
Jul 28 22:09:59.380818 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.380768    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:09:59.404441 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.403915    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal" event={"ID":"ace7f39d4fac06f2546ed5091afdf977","Type":"ContainerStarted","Data":"c5f5789516f27d7ce7a913dc55b77ab6c9a1127b609abd89916b93c2216531f1"}
Jul 28 22:09:59.405122 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:09:59.405087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal" event={"ID":"ada9d69552da923316c419a39e053e4a","Type":"ContainerStarted","Data":"5a1958ba3b55018fc53f8b5bcea157f002464f2519ca12de261d0b3d057f38ac"}
Jul 28 22:10:00.169965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.169779    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:10:00.200631 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.200396    2560 apiserver.go:52] "Watching apiserver"
Jul 28 22:10:00.207669 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.207611    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 22:10:00.209500 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.209464    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-htn4s","openshift-network-operator/iptables-alerter-wmczj","openshift-ovn-kubernetes/ovnkube-node-xphqh","kube-system/global-pull-secret-syncer-4dpq8","kube-system/konnectivity-agent-6frnt","openshift-cluster-node-tuning-operator/tuned-rzj9p","openshift-dns/node-resolver-4prbw","openshift-image-registry/node-ca-x75qb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal","openshift-multus/network-metrics-daemon-kdlqb","openshift-network-diagnostics/network-check-target-th9pq","kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28","openshift-multus/multus-additional-cni-plugins-hlbf4"]
Jul 28 22:10:00.211695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.211297    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.212681 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.212510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.214180 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.213948    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 22:10:00.214317 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.214263    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 22:10:00.214397 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.214319    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 22:10:00.214829 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.214534    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 22:10:00.214829 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.214634    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l9c5p\""
Jul 28 22:10:00.215013 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.214879    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.215640 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.215616    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v48n8\""
Jul 28 22:10:00.216492 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.216288    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 22:10:00.216492 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.216385    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 22:10:00.216492 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.216394    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 22:10:00.216928 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.216911    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.216998 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.216979    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:00.218845 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.218643    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 22:10:00.218845 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.218823    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dph6m\""
Jul 28 22:10:00.219037 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.218920    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 22:10:00.219037 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.218948    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 22:10:00.219154 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.219109    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 22:10:00.219154 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.219117    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 22:10:00.219321 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.219152    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 22:10:00.220253 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.220178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:00.220849 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.220832    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 22:10:00.221624 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.221604    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.223339 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.223316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.225084 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.224757    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 22:10:00.225084 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.224837    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 22:10:00.225084 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.224852    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 22:10:00.225084 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.224857    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ggdns\""
Jul 28 22:10:00.225084 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.224887    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n4jnw\""
Jul 28 22:10:00.225395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.225155    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 22:10:00.228721 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.225822    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 22:10:00.228721 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.226040    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.228721 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.226311    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gq5wl\""
Jul 28 22:10:00.228721 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.226319    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 22:10:00.228721 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.228150    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:00.228721 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.228348    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:00.229671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.229462    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 22:10:00.229671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.229476    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-62km7\""
Jul 28 22:10:00.229671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.229546    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 22:10:00.230271 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.229922    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 22:10:00.230271 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.230193    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:00.230271 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.230252    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:00.232540 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232520    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.232540 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94c1f799-00f0-4a61-b199-8db35aa6169c-dbus\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.232695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b043760-c4fb-44dc-b4fe-b5fef68a1068-agent-certs\") pod \"konnectivity-agent-6frnt\" (UID: \"1b043760-c4fb-44dc-b4fe-b5fef68a1068\") " pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:00.232695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-sys\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.232695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-conf-dir\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.232695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232641    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvsj\" (UniqueName: \"kubernetes.io/projected/1989b10c-dfaf-452a-a921-9af2e8537ca7-kube-api-access-tpvsj\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.232695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-run-systemd\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.232695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232690    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-cnibin\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232729    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e72a05ca-2762-47a0-b741-ef38c29dd48f-iptables-alerter-script\") pod \"iptables-alerter-wmczj\" (UID: \"e72a05ca-2762-47a0-b741-ef38c29dd48f\") " pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-etc-kubernetes\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232780    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-log-socket\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-os-release\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-hostroot\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-kubelet\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-run-openvswitch\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-tuned\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.232994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ae61615-faed-4cfa-8491-d7acc6d171ca-tmp\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-var-lib-cni-multus\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233048    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjbf\" (UniqueName: \"kubernetes.io/projected/e72a05ca-2762-47a0-b741-ef38c29dd48f-kube-api-access-fcjbf\") pod \"iptables-alerter-wmczj\" (UID: \"e72a05ca-2762-47a0-b741-ef38c29dd48f\") " pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.233086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233088    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-var-lib-kubelet\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-systemd-units\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-modprobe-d\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233186    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-kubernetes\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-systemd\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-socket-dir-parent\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-run-ovn-kubernetes\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b043760-c4fb-44dc-b4fe-b5fef68a1068-konnectivity-ca\") pod \"konnectivity-agent-6frnt\" (UID: \"1b043760-c4fb-44dc-b4fe-b5fef68a1068\") " pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-run\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-system-cni-dir\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-lib-modules\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-var-lib-kubelet\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-node-log\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233557    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-run-netns\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233582    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b1aabba-0081-45c1-a027-cdbae9442174-env-overrides\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b1aabba-0081-45c1-a027-cdbae9442174-ovn-node-metrics-cert\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.233752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-run-ovn\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-cni-bin\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-sysctl-conf\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1989b10c-dfaf-452a-a921-9af2e8537ca7-cni-binary-copy\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e72a05ca-2762-47a0-b741-ef38c29dd48f-host-slash\") pod \"iptables-alerter-wmczj\" (UID: \"e72a05ca-2762-47a0-b741-ef38c29dd48f\") " pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-slash\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233806    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b1aabba-0081-45c1-a027-cdbae9442174-ovnkube-script-lib\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233842    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-sysconfig\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdkt8\" (UniqueName: \"kubernetes.io/projected/5ae61615-faed-4cfa-8491-d7acc6d171ca-kube-api-access-rdkt8\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-run-multus-certs\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233933    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-etc-openvswitch\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b1aabba-0081-45c1-a027-cdbae9442174-ovnkube-config\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.233998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-sysctl-d\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-run-k8s-cni-cncf-io\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234068    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-daemon-config\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th8dn\" (UniqueName: \"kubernetes.io/projected/1b1aabba-0081-45c1-a027-cdbae9442174-kube-api-access-th8dn\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.234579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94c1f799-00f0-4a61-b199-8db35aa6169c-kubelet-config\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.235353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-host\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.235353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-run-netns\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.235353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-cni-netd\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.235353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-cni-dir\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.235353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234292    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-var-lib-cni-bin\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.235353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234321    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-var-lib-openvswitch\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.235353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.235353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.234959    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 22:10:00.235353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.235011    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.235897 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.235572    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 22:10:00.235897 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.235595    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wqphj\""
Jul 28 22:10:00.235897 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.235617    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 22:10:00.236066 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.236037    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 22:10:00.237736 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.237717    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 22:10:00.237966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.237942    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 22:10:00.238131 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.238112    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pzrlj\""
Jul 28 22:10:00.259585 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.259558    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzwlp"
Jul 28 22:10:00.266184 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.266102    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzwlp"
Jul 28 22:10:00.322181 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.322158    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 22:10:00.334596 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-run\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.334596 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-system-cni-dir\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994w6\" (UniqueName: \"kubernetes.io/projected/ba035fdb-cc39-4bfa-aa6b-81d417c51ad1-kube-api-access-994w6\") pod \"node-resolver-4prbw\" (UID: \"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1\") " pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-lib-modules\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-var-lib-kubelet\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-node-log\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334707    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-system-cni-dir\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-run-netns\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334756    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b1aabba-0081-45c1-a027-cdbae9442174-env-overrides\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334776    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-run\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334796    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-var-lib-kubelet\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.334853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b1aabba-0081-45c1-a027-cdbae9442174-ovn-node-metrics-cert\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334872    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-os-release\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnw2\" (UniqueName: \"kubernetes.io/projected/8c4023fd-ac12-4f44-ba1d-51693e370007-kube-api-access-xpnw2\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.334956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba035fdb-cc39-4bfa-aa6b-81d417c51ad1-tmp-dir\") pod \"node-resolver-4prbw\" (UID: \"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1\") " pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335040    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-node-log\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-run-netns\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-run-ovn\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-cni-bin\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-sysctl-conf\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-lib-modules\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1989b10c-dfaf-452a-a921-9af2e8537ca7-cni-binary-copy\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e72a05ca-2762-47a0-b741-ef38c29dd48f-host-slash\") pod \"iptables-alerter-wmczj\" (UID: \"e72a05ca-2762-47a0-b741-ef38c29dd48f\") " pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-run-ovn\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.335395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b1aabba-0081-45c1-a027-cdbae9442174-env-overrides\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335424    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-slash\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335424    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e72a05ca-2762-47a0-b741-ef38c29dd48f-host-slash\") pod \"iptables-alerter-wmczj\" (UID: \"e72a05ca-2762-47a0-b741-ef38c29dd48f\") " pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b1aabba-0081-45c1-a027-cdbae9442174-ovnkube-script-lib\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-cni-bin\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-sysconfig\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdkt8\" (UniqueName: \"kubernetes.io/projected/5ae61615-faed-4cfa-8491-d7acc6d171ca-kube-api-access-rdkt8\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-run-multus-certs\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-etc-openvswitch\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b1aabba-0081-45c1-a027-cdbae9442174-ovnkube-config\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-slash\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.335748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szfqx\" (UniqueName: \"kubernetes.io/projected/28de00f9-d2c8-4535-b6aa-29296c3ae000-kube-api-access-szfqx\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:00.336228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.336224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:00.336761 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.336276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-sysctl-d\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.336761 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.336731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1989b10c-dfaf-452a-a921-9af2e8537ca7-cni-binary-copy\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.336877 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.336857    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-run-k8s-cni-cncf-io\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.337001 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.336979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-sysctl-d\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.337356 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.337323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-run-multus-certs\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.337910 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.337887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b1aabba-0081-45c1-a027-cdbae9442174-ovnkube-config\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.337998 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.337954    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-etc-openvswitch\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.338165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.338146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b1aabba-0081-45c1-a027-cdbae9442174-ovnkube-script-lib\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.338309 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.338291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-sysctl-conf\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.338457 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.338434    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-sysconfig\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.338659 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.338629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-run-k8s-cni-cncf-io\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.338748 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.338685    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-daemon-config\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.338748 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.338717    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th8dn\" (UniqueName: \"kubernetes.io/projected/1b1aabba-0081-45c1-a027-cdbae9442174-kube-api-access-th8dn\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.339041 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.338889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-registration-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.339041 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.338938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-system-cni-dir\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.339041 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.338973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94c1f799-00f0-4a61-b199-8db35aa6169c-kubelet-config\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.339041 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-host\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.339041 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-run-netns\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.339378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-cni-netd\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.339378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339118    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59x6\" (UniqueName: \"kubernetes.io/projected/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-kube-api-access-p59x6\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.339378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-cni-dir\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.339378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-var-lib-cni-bin\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.339378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339287    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-daemon-config\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.339378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-var-lib-openvswitch\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94c1f799-00f0-4a61-b199-8db35aa6169c-kubelet-config\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-cnibin\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339469    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-host\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94c1f799-00f0-4a61-b199-8db35aa6169c-dbus\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339514    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b043760-c4fb-44dc-b4fe-b5fef68a1068-agent-certs\") pod \"konnectivity-agent-6frnt\" (UID: \"1b043760-c4fb-44dc-b4fe-b5fef68a1068\") " pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-run-netns\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-sys\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339595    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-conf-dir\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339634    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvsj\" (UniqueName: \"kubernetes.io/projected/1989b10c-dfaf-452a-a921-9af2e8537ca7-kube-api-access-tpvsj\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-run-systemd\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-socket-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.339701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-var-lib-cni-bin\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339728    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-device-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-cnibin\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94c1f799-00f0-4a61-b199-8db35aa6169c-dbus\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-cni-dir\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-cni-netd\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e72a05ca-2762-47a0-b741-ef38c29dd48f-iptables-alerter-script\") pod \"iptables-alerter-wmczj\" (UID: \"e72a05ca-2762-47a0-b741-ef38c29dd48f\") " pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-etc-kubernetes\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.339887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/389f391a-1877-421e-932f-2dae44bf6738-host\") pod \"node-ca-x75qb\" (UID: \"389f391a-1877-421e-932f-2dae44bf6738\") " pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-cnibin\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340136    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-sys\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340189    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-etc-kubernetes\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-var-lib-openvswitch\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340265    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-conf-dir\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.340377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-run-systemd\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e72a05ca-2762-47a0-b741-ef38c29dd48f-iptables-alerter-script\") pod \"iptables-alerter-wmczj\" (UID: \"e72a05ca-2762-47a0-b741-ef38c29dd48f\") " pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlh2\" (UniqueName: \"kubernetes.io/projected/389f391a-1877-421e-932f-2dae44bf6738-kube-api-access-jvlh2\") pod \"node-ca-x75qb\" (UID: \"389f391a-1877-421e-932f-2dae44bf6738\") " pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c4023fd-ac12-4f44-ba1d-51693e370007-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c4023fd-ac12-4f44-ba1d-51693e370007-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-log-socket\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-os-release\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340797    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-hostroot\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340812    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-os-release\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-kubelet\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-run-openvswitch\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c4023fd-ac12-4f44-ba1d-51693e370007-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.340990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-tuned\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.341027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ae61615-faed-4cfa-8491-d7acc6d171ca-tmp\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.341065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-var-lib-cni-multus\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.341227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-var-lib-cni-multus\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.341327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.341277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-run-openvswitch\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.342312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.341319    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-log-socket\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.342312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.341355    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-hostroot\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.342312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.341396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-kubelet\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.342312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.341707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjbf\" (UniqueName: \"kubernetes.io/projected/e72a05ca-2762-47a0-b741-ef38c29dd48f-kube-api-access-fcjbf\") pod \"iptables-alerter-wmczj\" (UID: \"e72a05ca-2762-47a0-b741-ef38c29dd48f\") " pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.342312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342074    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b1aabba-0081-45c1-a027-cdbae9442174-ovn-node-metrics-cert\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.342312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba035fdb-cc39-4bfa-aa6b-81d417c51ad1-hosts-file\") pod \"node-resolver-4prbw\" (UID: \"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1\") " pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.342312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-var-lib-kubelet\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-systemd-units\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342358    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/389f391a-1877-421e-932f-2dae44bf6738-serviceca\") pod \"node-ca-x75qb\" (UID: \"389f391a-1877-421e-932f-2dae44bf6738\") " pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-sys-fs\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342415    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-modprobe-d\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-kubernetes\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342525    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-systemd\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-socket-dir-parent\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342588    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-run-ovn-kubernetes\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.342671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.342649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b043760-c4fb-44dc-b4fe-b5fef68a1068-konnectivity-ca\") pod \"konnectivity-agent-6frnt\" (UID: \"1b043760-c4fb-44dc-b4fe-b5fef68a1068\") " pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:00.343794 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.343454    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b043760-c4fb-44dc-b4fe-b5fef68a1068-agent-certs\") pod \"konnectivity-agent-6frnt\" (UID: \"1b043760-c4fb-44dc-b4fe-b5fef68a1068\") " pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:00.343794 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.343588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-kubernetes\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.343794 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.343632    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:00.343794 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.343696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-modprobe-d\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.343794 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.343727    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret podName:94c1f799-00f0-4a61-b199-8db35aa6169c nodeName:}" failed. No retries permitted until 2026-07-28 22:10:00.843698487 +0000 UTC m=+2.966046372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret") pod "global-pull-secret-syncer-4dpq8" (UID: "94c1f799-00f0-4a61-b199-8db35aa6169c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:00.343794 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.343763    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-host-var-lib-kubelet\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.344206 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.343867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-host-run-ovn-kubernetes\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.344206 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.343903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-systemd\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.344206 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.343989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b1aabba-0081-45c1-a027-cdbae9442174-systemd-units\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.344206 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.344021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1989b10c-dfaf-452a-a921-9af2e8537ca7-multus-socket-dir-parent\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.344454 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.344434    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b043760-c4fb-44dc-b4fe-b5fef68a1068-konnectivity-ca\") pod \"konnectivity-agent-6frnt\" (UID: \"1b043760-c4fb-44dc-b4fe-b5fef68a1068\") " pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:00.344521 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.344453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ae61615-faed-4cfa-8491-d7acc6d171ca-tmp\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.346026 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.345998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ae61615-faed-4cfa-8491-d7acc6d171ca-etc-tuned\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.346907 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.346886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdkt8\" (UniqueName: \"kubernetes.io/projected/5ae61615-faed-4cfa-8491-d7acc6d171ca-kube-api-access-rdkt8\") pod \"tuned-rzj9p\" (UID: \"5ae61615-faed-4cfa-8491-d7acc6d171ca\") " pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.348499 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.348478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th8dn\" (UniqueName: \"kubernetes.io/projected/1b1aabba-0081-45c1-a027-cdbae9442174-kube-api-access-th8dn\") pod \"ovnkube-node-xphqh\" (UID: \"1b1aabba-0081-45c1-a027-cdbae9442174\") " pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.349140 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.349123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvsj\" (UniqueName: \"kubernetes.io/projected/1989b10c-dfaf-452a-a921-9af2e8537ca7-kube-api-access-tpvsj\") pod \"multus-htn4s\" (UID: \"1989b10c-dfaf-452a-a921-9af2e8537ca7\") " pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.357393 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.357367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjbf\" (UniqueName: \"kubernetes.io/projected/e72a05ca-2762-47a0-b741-ef38c29dd48f-kube-api-access-fcjbf\") pod \"iptables-alerter-wmczj\" (UID: \"e72a05ca-2762-47a0-b741-ef38c29dd48f\") " pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.443531 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-os-release\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.443531 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnw2\" (UniqueName: \"kubernetes.io/projected/8c4023fd-ac12-4f44-ba1d-51693e370007-kube-api-access-xpnw2\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.443531 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba035fdb-cc39-4bfa-aa6b-81d417c51ad1-tmp-dir\") pod \"node-resolver-4prbw\" (UID: \"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1\") " pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.443531 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443627    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-os-release\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szfqx\" (UniqueName: \"kubernetes.io/projected/28de00f9-d2c8-4535-b6aa-29296c3ae000-kube-api-access-szfqx\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-registration-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-system-cni-dir\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443767    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p59x6\" (UniqueName: \"kubernetes.io/projected/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-kube-api-access-p59x6\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-cnibin\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-socket-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.443882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-registration-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-device-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443897    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-system-cni-dir\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443924    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/389f391a-1877-421e-932f-2dae44bf6738-host\") pod \"node-ca-x75qb\" (UID: \"389f391a-1877-421e-932f-2dae44bf6738\") " pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlh2\" (UniqueName: \"kubernetes.io/projected/389f391a-1877-421e-932f-2dae44bf6738-kube-api-access-jvlh2\") pod \"node-ca-x75qb\" (UID: \"389f391a-1877-421e-932f-2dae44bf6738\") " pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443953    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba035fdb-cc39-4bfa-aa6b-81d417c51ad1-tmp-dir\") pod \"node-resolver-4prbw\" (UID: \"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1\") " pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443977    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-socket-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-device-dir\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.443999    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c4023fd-ac12-4f44-ba1d-51693e370007-cnibin\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/389f391a-1877-421e-932f-2dae44bf6738-host\") pod \"node-ca-x75qb\" (UID: \"389f391a-1877-421e-932f-2dae44bf6738\") " pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c4023fd-ac12-4f44-ba1d-51693e370007-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c4023fd-ac12-4f44-ba1d-51693e370007-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c4023fd-ac12-4f44-ba1d-51693e370007-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba035fdb-cc39-4bfa-aa6b-81d417c51ad1-hosts-file\") pod \"node-resolver-4prbw\" (UID: \"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1\") " pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/389f391a-1877-421e-932f-2dae44bf6738-serviceca\") pod \"node-ca-x75qb\" (UID: \"389f391a-1877-421e-932f-2dae44bf6738\") " pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.444513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-sys-fs\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.445204 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba035fdb-cc39-4bfa-aa6b-81d417c51ad1-hosts-file\") pod \"node-resolver-4prbw\" (UID: \"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1\") " pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.445204 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994w6\" (UniqueName: \"kubernetes.io/projected/ba035fdb-cc39-4bfa-aa6b-81d417c51ad1-kube-api-access-994w6\") pod \"node-resolver-4prbw\" (UID: \"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1\") " pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.445204 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.444467    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:00.445204 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.444543    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs podName:28de00f9-d2c8-4535-b6aa-29296c3ae000 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:00.94452183 +0000 UTC m=+3.066869709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs") pod "network-metrics-daemon-kdlqb" (UID: "28de00f9-d2c8-4535-b6aa-29296c3ae000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:00.445204 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-sys-fs\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.445204 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.444697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/389f391a-1877-421e-932f-2dae44bf6738-serviceca\") pod \"node-ca-x75qb\" (UID: \"389f391a-1877-421e-932f-2dae44bf6738\") " pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.445561 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.445424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c4023fd-ac12-4f44-ba1d-51693e370007-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.445561 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.445429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c4023fd-ac12-4f44-ba1d-51693e370007-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.445561 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.445445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c4023fd-ac12-4f44-ba1d-51693e370007-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.447846 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.447828    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:10:00.453740 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.453714    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:10:00.453740 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.453743    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:10:00.453921 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.453759    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hxbnj for pod openshift-network-diagnostics/network-check-target-th9pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:00.453921 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.453868    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj podName:4e8133ea-cd7d-4ee6-9d82-86fc41d0d532 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:00.953850557 +0000 UTC m=+3.076198435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hxbnj" (UniqueName: "kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj") pod "network-check-target-th9pq" (UID: "4e8133ea-cd7d-4ee6-9d82-86fc41d0d532") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:00.456362 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.456341    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szfqx\" (UniqueName: \"kubernetes.io/projected/28de00f9-d2c8-4535-b6aa-29296c3ae000-kube-api-access-szfqx\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:00.456858 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.456826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59x6\" (UniqueName: \"kubernetes.io/projected/6594c0a4-33f6-4633-afbb-3e3e50c81bf2-kube-api-access-p59x6\") pod \"aws-ebs-csi-driver-node-vhn28\" (UID: \"6594c0a4-33f6-4633-afbb-3e3e50c81bf2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.456961 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.456876    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-994w6\" (UniqueName: \"kubernetes.io/projected/ba035fdb-cc39-4bfa-aa6b-81d417c51ad1-kube-api-access-994w6\") pod \"node-resolver-4prbw\" (UID: \"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1\") " pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.457215 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.457191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnw2\" (UniqueName: \"kubernetes.io/projected/8c4023fd-ac12-4f44-ba1d-51693e370007-kube-api-access-xpnw2\") pod \"multus-additional-cni-plugins-hlbf4\" (UID: \"8c4023fd-ac12-4f44-ba1d-51693e370007\") " pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.457462 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.457438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlh2\" (UniqueName: \"kubernetes.io/projected/389f391a-1877-421e-932f-2dae44bf6738-kube-api-access-jvlh2\") pod \"node-ca-x75qb\" (UID: \"389f391a-1877-421e-932f-2dae44bf6738\") " pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.469069 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.469046    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:10:00.525668 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.525634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-htn4s"
Jul 28 22:10:00.537243 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.537215    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wmczj"
Jul 28 22:10:00.546668 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.546641    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:00.562192 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.562171    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:00.571674 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.571649    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rzj9p"
Jul 28 22:10:00.580218 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.580187    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4prbw"
Jul 28 22:10:00.587534 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.587508    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x75qb"
Jul 28 22:10:00.596287 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.596266    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28"
Jul 28 22:10:00.603776 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.603756    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hlbf4"
Jul 28 22:10:00.846678 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.846587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:00.846836 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.846733    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:00.846836 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.846805    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret podName:94c1f799-00f0-4a61-b199-8db35aa6169c nodeName:}" failed. No retries permitted until 2026-07-28 22:10:01.846787488 +0000 UTC m=+3.969135360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret") pod "global-pull-secret-syncer-4dpq8" (UID: "94c1f799-00f0-4a61-b199-8db35aa6169c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:00.947319 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:00.947269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:00.947521 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.947392    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:00.947521 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:00.947474    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs podName:28de00f9-d2c8-4535-b6aa-29296c3ae000 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:01.947452565 +0000 UTC m=+4.069800459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs") pod "network-metrics-daemon-kdlqb" (UID: "28de00f9-d2c8-4535-b6aa-29296c3ae000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:01.048619 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:01.048583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:01.048782 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:01.048727    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:10:01.048782 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:01.048743    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:10:01.048782 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:01.048754    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hxbnj for pod openshift-network-diagnostics/network-check-target-th9pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:01.048924 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:01.048810    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj podName:4e8133ea-cd7d-4ee6-9d82-86fc41d0d532 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:02.048795375 +0000 UTC m=+4.171143248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxbnj" (UniqueName: "kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj") pod "network-check-target-th9pq" (UID: "4e8133ea-cd7d-4ee6-9d82-86fc41d0d532") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:01.268480 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:01.268395    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 22:05:00 +0000 UTC" deadline="2028-02-03 14:28:05.426005147 +0000 UTC"
Jul 28 22:10:01.268480 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:01.268423    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13312h18m4.157585014s"
Jul 28 22:10:01.401323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:01.401298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:01.401447 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:01.401424    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:01.408347 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:01.408056    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6594c0a4_33f6_4633_afbb_3e3e50c81bf2.slice/crio-97dcbba90a29fa602302e7d0c96652f596669b7e59f2250d1dd283582b311e14 WatchSource:0}: Error finding container 97dcbba90a29fa602302e7d0c96652f596669b7e59f2250d1dd283582b311e14: Status 404 returned error can't find the container with id 97dcbba90a29fa602302e7d0c96652f596669b7e59f2250d1dd283582b311e14
Jul 28 22:10:01.411032 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:01.411006    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1aabba_0081_45c1_a027_cdbae9442174.slice/crio-2be74480abdce2745201f872876249463f81c561ca89afebd9575ce549d4761a WatchSource:0}: Error finding container 2be74480abdce2745201f872876249463f81c561ca89afebd9575ce549d4761a: Status 404 returned error can't find the container with id 2be74480abdce2745201f872876249463f81c561ca89afebd9575ce549d4761a
Jul 28 22:10:01.412291 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:01.412248    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b043760_c4fb_44dc_b4fe_b5fef68a1068.slice/crio-58f815912f87e1fd2b474f533f0bc6f86a8f7c5c686f93914c0d89623b0daa91 WatchSource:0}: Error finding container 58f815912f87e1fd2b474f533f0bc6f86a8f7c5c686f93914c0d89623b0daa91: Status 404 returned error can't find the container with id 58f815912f87e1fd2b474f533f0bc6f86a8f7c5c686f93914c0d89623b0daa91
Jul 28 22:10:01.413829 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:01.413798    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1989b10c_dfaf_452a_a921_9af2e8537ca7.slice/crio-82a9a82c4dd5787a0fa55c857481f10d10afb122514c0646951822f16b184f32 WatchSource:0}: Error finding container 82a9a82c4dd5787a0fa55c857481f10d10afb122514c0646951822f16b184f32: Status 404 returned error can't find the container with id 82a9a82c4dd5787a0fa55c857481f10d10afb122514c0646951822f16b184f32
Jul 28 22:10:01.414871 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:01.414846    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae61615_faed_4cfa_8491_d7acc6d171ca.slice/crio-896a7622a824ea5c4e4c7684d7184eef8697d3222569e123f9592dcf562a93c7 WatchSource:0}: Error finding container 896a7622a824ea5c4e4c7684d7184eef8697d3222569e123f9592dcf562a93c7: Status 404 returned error can't find the container with id 896a7622a824ea5c4e4c7684d7184eef8697d3222569e123f9592dcf562a93c7
Jul 28 22:10:01.416172 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:01.416139    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4023fd_ac12_4f44_ba1d_51693e370007.slice/crio-f85e4580639d973b66742a207dc55e1f8ff82657e0eb0832a3fd70e57415cff7 WatchSource:0}: Error finding container f85e4580639d973b66742a207dc55e1f8ff82657e0eb0832a3fd70e57415cff7: Status 404 returned error can't find the container with id f85e4580639d973b66742a207dc55e1f8ff82657e0eb0832a3fd70e57415cff7
Jul 28 22:10:01.418170 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:01.417756    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389f391a_1877_421e_932f_2dae44bf6738.slice/crio-b2b075f40fa4be880f6fcfbc970e434af34dceb6a8784098b3d317ffe530ceb0 WatchSource:0}: Error finding container b2b075f40fa4be880f6fcfbc970e434af34dceb6a8784098b3d317ffe530ceb0: Status 404 returned error can't find the container with id b2b075f40fa4be880f6fcfbc970e434af34dceb6a8784098b3d317ffe530ceb0
Jul 28 22:10:01.855803 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:01.855272    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:01.855803 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:01.855518    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:01.855803 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:01.855793    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret podName:94c1f799-00f0-4a61-b199-8db35aa6169c nodeName:}" failed. No retries permitted until 2026-07-28 22:10:03.855764941 +0000 UTC m=+5.978112831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret") pod "global-pull-secret-syncer-4dpq8" (UID: "94c1f799-00f0-4a61-b199-8db35aa6169c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:01.957153 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:01.956474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:01.957153 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:01.956710    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:01.957153 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:01.956783    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs podName:28de00f9-d2c8-4535-b6aa-29296c3ae000 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:03.956761598 +0000 UTC m=+6.079109478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs") pod "network-metrics-daemon-kdlqb" (UID: "28de00f9-d2c8-4535-b6aa-29296c3ae000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:02.058109 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.057428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:02.058109 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:02.057628    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:10:02.058109 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:02.057652    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:10:02.058109 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:02.057667    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hxbnj for pod openshift-network-diagnostics/network-check-target-th9pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:02.058109 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:02.057731    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj podName:4e8133ea-cd7d-4ee6-9d82-86fc41d0d532 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:04.057709471 +0000 UTC m=+6.180057371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxbnj" (UniqueName: "kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj") pod "network-check-target-th9pq" (UID: "4e8133ea-cd7d-4ee6-9d82-86fc41d0d532") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:02.269284 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.269210    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 22:05:00 +0000 UTC" deadline="2028-01-06 10:56:51.930758343 +0000 UTC"
Jul 28 22:10:02.269284 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.269281    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12636h46m49.661482091s"
Jul 28 22:10:02.401304 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.401275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:02.401491 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:02.401404    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:02.404046 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.404018    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:02.404201 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:02.404126    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:02.420828 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.420767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rzj9p" event={"ID":"5ae61615-faed-4cfa-8491-d7acc6d171ca","Type":"ContainerStarted","Data":"896a7622a824ea5c4e4c7684d7184eef8697d3222569e123f9592dcf562a93c7"}
Jul 28 22:10:02.424996 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.424961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" event={"ID":"1b1aabba-0081-45c1-a027-cdbae9442174","Type":"ContainerStarted","Data":"2be74480abdce2745201f872876249463f81c561ca89afebd9575ce549d4761a"}
Jul 28 22:10:02.432808 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.431948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal" event={"ID":"ace7f39d4fac06f2546ed5091afdf977","Type":"ContainerStarted","Data":"a94ffdff6e7b8a9c061a105b01f250518c87445c8fdda06050e680e046e01bb5"}
Jul 28 22:10:02.446872 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.446841    2560 generic.go:356] "Generic (PLEG): container finished" podID="ada9d69552da923316c419a39e053e4a" containerID="3cb4552e0b53b4b19b1d9d7ded5b0f5d76bfe5db167ada3a26096bed2665462a" exitCode=0
Jul 28 22:10:02.446997 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.446930    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal" event={"ID":"ada9d69552da923316c419a39e053e4a","Type":"ContainerDied","Data":"3cb4552e0b53b4b19b1d9d7ded5b0f5d76bfe5db167ada3a26096bed2665462a"}
Jul 28 22:10:02.466501 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.466234    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal" podStartSLOduration=3.466212658 podStartE2EDuration="3.466212658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:02.450369414 +0000 UTC m=+4.572717311" watchObservedRunningTime="2026-07-28 22:10:02.466212658 +0000 UTC m=+4.588560555"
Jul 28 22:10:02.473910 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.472132    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4prbw" event={"ID":"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1","Type":"ContainerStarted","Data":"bea369e3e38d5121fcab8e26a55a7127667b70fa840c5e6c469338dc1f96085d"}
Jul 28 22:10:02.494310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.494280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x75qb" event={"ID":"389f391a-1877-421e-932f-2dae44bf6738","Type":"ContainerStarted","Data":"b2b075f40fa4be880f6fcfbc970e434af34dceb6a8784098b3d317ffe530ceb0"}
Jul 28 22:10:02.520573 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.512338    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-htn4s" event={"ID":"1989b10c-dfaf-452a-a921-9af2e8537ca7","Type":"ContainerStarted","Data":"82a9a82c4dd5787a0fa55c857481f10d10afb122514c0646951822f16b184f32"}
Jul 28 22:10:02.520573 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.513397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6frnt" event={"ID":"1b043760-c4fb-44dc-b4fe-b5fef68a1068","Type":"ContainerStarted","Data":"58f815912f87e1fd2b474f533f0bc6f86a8f7c5c686f93914c0d89623b0daa91"}
Jul 28 22:10:02.520573 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.514737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28" event={"ID":"6594c0a4-33f6-4633-afbb-3e3e50c81bf2","Type":"ContainerStarted","Data":"97dcbba90a29fa602302e7d0c96652f596669b7e59f2250d1dd283582b311e14"}
Jul 28 22:10:02.520573 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.515913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wmczj" event={"ID":"e72a05ca-2762-47a0-b741-ef38c29dd48f","Type":"ContainerStarted","Data":"160885198ac17207c8d98c569b51bfc3d09ec563ccc1678869c1f392e065bb2d"}
Jul 28 22:10:02.520573 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:02.517013    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlbf4" event={"ID":"8c4023fd-ac12-4f44-ba1d-51693e370007","Type":"ContainerStarted","Data":"f85e4580639d973b66742a207dc55e1f8ff82657e0eb0832a3fd70e57415cff7"}
Jul 28 22:10:03.401586 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:03.401550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:03.402091 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:03.401705    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:03.538456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:03.538410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal" event={"ID":"ada9d69552da923316c419a39e053e4a","Type":"ContainerStarted","Data":"1f7f43e32de3cdbb5a6dbceb401f061fc4f4a3b34ceafcf820596e369c8100b9"}
Jul 28 22:10:03.555030 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:03.554970    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal" podStartSLOduration=4.554947932 podStartE2EDuration="4.554947932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:03.553943528 +0000 UTC m=+5.676291427" watchObservedRunningTime="2026-07-28 22:10:03.554947932 +0000 UTC m=+5.677295830"
Jul 28 22:10:03.875463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:03.874801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:03.875463 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:03.875012    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:03.875463 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:03.875079    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret podName:94c1f799-00f0-4a61-b199-8db35aa6169c nodeName:}" failed. No retries permitted until 2026-07-28 22:10:07.87505944 +0000 UTC m=+9.997407316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret") pod "global-pull-secret-syncer-4dpq8" (UID: "94c1f799-00f0-4a61-b199-8db35aa6169c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:03.975385 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:03.975343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:03.975580 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:03.975514    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:03.975646 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:03.975582    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs podName:28de00f9-d2c8-4535-b6aa-29296c3ae000 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:07.975561965 +0000 UTC m=+10.097909858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs") pod "network-metrics-daemon-kdlqb" (UID: "28de00f9-d2c8-4535-b6aa-29296c3ae000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:04.076760 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:04.076712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:04.076974 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:04.076951    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:10:04.077059 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:04.076978    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:10:04.077059 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:04.076999    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hxbnj for pod openshift-network-diagnostics/network-check-target-th9pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:04.077188 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:04.077066    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj podName:4e8133ea-cd7d-4ee6-9d82-86fc41d0d532 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:08.077044326 +0000 UTC m=+10.199392225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxbnj" (UniqueName: "kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj") pod "network-check-target-th9pq" (UID: "4e8133ea-cd7d-4ee6-9d82-86fc41d0d532") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:04.402290 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:04.402254    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:04.402792 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:04.402308    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:04.402792 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:04.402412    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:04.402792 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:04.402543    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:05.401182 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:05.401144    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:05.401360 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:05.401293    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:06.400920 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:06.400886    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:06.401415 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:06.400885    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:06.401415 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:06.401033    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:06.401415 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:06.401189    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:07.401159 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:07.401125    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:07.401601 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:07.401264    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:07.910079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:07.910044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:07.910278 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:07.910214    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:07.910344 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:07.910288    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret podName:94c1f799-00f0-4a61-b199-8db35aa6169c nodeName:}" failed. No retries permitted until 2026-07-28 22:10:15.910266942 +0000 UTC m=+18.032614832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret") pod "global-pull-secret-syncer-4dpq8" (UID: "94c1f799-00f0-4a61-b199-8db35aa6169c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:08.011352 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:08.011323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:08.011480 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:08.011455    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:08.011519 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:08.011509    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs podName:28de00f9-d2c8-4535-b6aa-29296c3ae000 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:16.011493237 +0000 UTC m=+18.133841114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs") pod "network-metrics-daemon-kdlqb" (UID: "28de00f9-d2c8-4535-b6aa-29296c3ae000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:08.112302 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:08.112264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:08.112467 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:08.112453    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:10:08.112545 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:08.112475    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:10:08.112545 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:08.112489    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hxbnj for pod openshift-network-diagnostics/network-check-target-th9pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:08.112637 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:08.112555    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj podName:4e8133ea-cd7d-4ee6-9d82-86fc41d0d532 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:16.112534837 +0000 UTC m=+18.234882710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxbnj" (UniqueName: "kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj") pod "network-check-target-th9pq" (UID: "4e8133ea-cd7d-4ee6-9d82-86fc41d0d532") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:08.402455 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:08.401909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:08.402455 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:08.402024    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:08.402455 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:08.402420    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:08.403037 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:08.402512    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:09.401644 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:09.401602    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:09.401829 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:09.401771    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:10.401404 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:10.401369    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:10.401849 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:10.401503    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:10.401849 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:10.401613    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:10.401849 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:10.401753    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:11.401316 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:11.401282    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:11.401491 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:11.401428    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:12.400809 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:12.400777    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:12.400999 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:12.400907    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:12.400999 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:12.400993    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:12.401135 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:12.401083    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:13.401075 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:13.401041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:13.401509 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:13.401196    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:14.401464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:14.401432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:14.401924 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:14.401433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:14.401924 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:14.401580    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:14.401924 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:14.401611    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:15.401247 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:15.401206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:15.401482 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:15.401340    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:15.975238 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:15.975208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:15.975626 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:15.975325    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:15.975626 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:15.975375    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret podName:94c1f799-00f0-4a61-b199-8db35aa6169c nodeName:}" failed. No retries permitted until 2026-07-28 22:10:31.975360657 +0000 UTC m=+34.097708528 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret") pod "global-pull-secret-syncer-4dpq8" (UID: "94c1f799-00f0-4a61-b199-8db35aa6169c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:16.076050 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:16.075980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:16.076195 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:16.076140    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:16.076235 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:16.076195    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs podName:28de00f9-d2c8-4535-b6aa-29296c3ae000 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:32.076181159 +0000 UTC m=+34.198529034 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs") pod "network-metrics-daemon-kdlqb" (UID: "28de00f9-d2c8-4535-b6aa-29296c3ae000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:16.176486 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:16.176456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:16.176635 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:16.176572    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:10:16.176635 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:16.176588    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:10:16.176635 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:16.176596    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hxbnj for pod openshift-network-diagnostics/network-check-target-th9pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:16.176737 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:16.176642    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj podName:4e8133ea-cd7d-4ee6-9d82-86fc41d0d532 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:32.176629479 +0000 UTC m=+34.298977350 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxbnj" (UniqueName: "kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj") pod "network-check-target-th9pq" (UID: "4e8133ea-cd7d-4ee6-9d82-86fc41d0d532") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:16.401569 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:16.401492    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:16.401731 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:16.401494    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:16.401731 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:16.401605    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:16.401731 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:16.401670    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:17.400895 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:17.400858    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:17.401385 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:17.400979    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:18.402300 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:18.402266    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:18.402729 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:18.402362    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:18.402729 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:18.402442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:18.402729 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:18.402534    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:19.401148 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:19.401115    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:19.401316 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:19.401237    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:20.401446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.401278    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:20.401908 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:20.401560    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:20.401908 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.401718    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:20.401908 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:20.401808    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:20.576676 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.576474    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c4023fd-ac12-4f44-ba1d-51693e370007" containerID="64bce4738d466555d2d00fc3450e44327a42cfd80b5b5dd0059f9c3c411e9bb2" exitCode=0
Jul 28 22:10:20.576770 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.576550    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlbf4" event={"ID":"8c4023fd-ac12-4f44-ba1d-51693e370007","Type":"ContainerDied","Data":"64bce4738d466555d2d00fc3450e44327a42cfd80b5b5dd0059f9c3c411e9bb2"}
Jul 28 22:10:20.578306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.578280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rzj9p" event={"ID":"5ae61615-faed-4cfa-8491-d7acc6d171ca","Type":"ContainerStarted","Data":"5aecc7856c0d0a0fb799114d1b85a825032d2e37f31356d9df80a06363ec7932"}
Jul 28 22:10:20.583533 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.583506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" event={"ID":"1b1aabba-0081-45c1-a027-cdbae9442174","Type":"ContainerStarted","Data":"c54a67173b2f8a38d784b8954f43e7b05dcfc6b2b425a2184c78e8cc16855081"}
Jul 28 22:10:20.583641 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.583611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" event={"ID":"1b1aabba-0081-45c1-a027-cdbae9442174","Type":"ContainerStarted","Data":"58f4359d2ca760a1b2fbcffe1f3f457cf7ab7ff10c977c65abb17bc8ad58b4e0"}
Jul 28 22:10:20.583703 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.583663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" event={"ID":"1b1aabba-0081-45c1-a027-cdbae9442174","Type":"ContainerStarted","Data":"f5846ea16de2228f4dccddb2e20ffd7f1222cfb7643d34b229019af743b094f4"}
Jul 28 22:10:20.584942 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.584922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4prbw" event={"ID":"ba035fdb-cc39-4bfa-aa6b-81d417c51ad1","Type":"ContainerStarted","Data":"ea6568d3573bd95c0a65915524e66662adff11e85413fa7003d2993635e068ce"}
Jul 28 22:10:20.586415 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.586391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x75qb" event={"ID":"389f391a-1877-421e-932f-2dae44bf6738","Type":"ContainerStarted","Data":"aed3a5dbbf808fe9ec6a6eaa37a18448f074e7fc00b512d4e89b005e19372b61"}
Jul 28 22:10:20.587677 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.587658    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-htn4s" event={"ID":"1989b10c-dfaf-452a-a921-9af2e8537ca7","Type":"ContainerStarted","Data":"2704f8a713f2b6be7b4be2e76673b5d44fd10b599a48d130742611b0ef7f714d"}
Jul 28 22:10:20.588900 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.588878    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6frnt" event={"ID":"1b043760-c4fb-44dc-b4fe-b5fef68a1068","Type":"ContainerStarted","Data":"bdeac3955acd70ec8a8a6d1ddbe9183fe03affbed500905658d3e1b7a67733da"}
Jul 28 22:10:20.590197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.590176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28" event={"ID":"6594c0a4-33f6-4633-afbb-3e3e50c81bf2","Type":"ContainerStarted","Data":"65077e7d6b07e6bf7d762ce5cecfd3115c00335e7e72f0a04a9ce359ad2ca6aa"}
Jul 28 22:10:20.614111 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.614054    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6frnt" podStartSLOduration=2.907612637 podStartE2EDuration="21.614036013s" totalImagesPullingTime="18.706423376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:20.613314176 +0000 UTC m=+22.735662069" watchObservedRunningTime="2026-07-28 22:10:20.614036013 +0000 UTC m=+22.736383907"
Jul 28 22:10:20.628193 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.628151    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rzj9p" podStartSLOduration=2.872755753 podStartE2EDuration="21.628138506s" totalImagesPullingTime="18.755382753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:20.627966409 +0000 UTC m=+22.750314302" watchObservedRunningTime="2026-07-28 22:10:20.628138506 +0000 UTC m=+22.750486399"
Jul 28 22:10:20.641718 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.641679    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4prbw" podStartSLOduration=2.886616377 podStartE2EDuration="21.641665042s" totalImagesPullingTime="18.755048665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:20.641179294 +0000 UTC m=+22.763527187" watchObservedRunningTime="2026-07-28 22:10:20.641665042 +0000 UTC m=+22.764012935"
Jul 28 22:10:20.655759 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.655716    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x75qb" podStartSLOduration=7.260509687 podStartE2EDuration="21.655702562s" totalImagesPullingTime="14.395192875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:20.655243386 +0000 UTC m=+22.777591279" watchObservedRunningTime="2026-07-28 22:10:20.655702562 +0000 UTC m=+22.778050486"
Jul 28 22:10:20.671874 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.671830    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-htn4s" podStartSLOduration=2.858433043 podStartE2EDuration="21.671816919s" totalImagesPullingTime="18.813383876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:20.67150553 +0000 UTC m=+22.793853426" watchObservedRunningTime="2026-07-28 22:10:20.671816919 +0000 UTC m=+22.794164836"
Jul 28 22:10:20.726448 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.726418    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:20.822175 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.822145    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:20.922382 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:20.922350    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:21.354819 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.354786    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:21.355450 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.355433    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:21.401689 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.401660    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:21.402371 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:21.401805    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:21.596558 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.596526    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wmczj" event={"ID":"e72a05ca-2762-47a0-b741-ef38c29dd48f","Type":"ContainerStarted","Data":"28d8400e2fa6390fa86b3de44d0165c1c8d959e842f6eec754a32bd26bc5e7c3"}
Jul 28 22:10:21.601065 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.600950    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" event={"ID":"1b1aabba-0081-45c1-a027-cdbae9442174","Type":"ContainerStarted","Data":"e0406925296812fac2a0938fe54426b26ff21e3935f93cd5c638c7ff7840e3c8"}
Jul 28 22:10:21.601215 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.601069    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" event={"ID":"1b1aabba-0081-45c1-a027-cdbae9442174","Type":"ContainerStarted","Data":"1787c5ef853190067e27136330dcb5c70c77e4cb3369c14589d41efd5110ddc0"}
Jul 28 22:10:21.601215 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.601140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" event={"ID":"1b1aabba-0081-45c1-a027-cdbae9442174","Type":"ContainerStarted","Data":"33bdf6c9ff9c78911851225e305c987874a55ff984e292b073511572648533ae"}
Jul 28 22:10:21.601668 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.601647    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:21.601928 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.601909    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6frnt"
Jul 28 22:10:21.611403 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.611321    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wmczj" podStartSLOduration=3.864954214 podStartE2EDuration="22.611303773s" totalImagesPullingTime="18.746349559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:21.610654216 +0000 UTC m=+23.733002109" watchObservedRunningTime="2026-07-28 22:10:21.611303773 +0000 UTC m=+23.733651670"
Jul 28 22:10:21.635994 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.635970    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 22:10:21.730518 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.730483    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:21.811855 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.811815    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:21.871212 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:21.871095    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:22.295084 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:22.294989    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T22:10:21.635987515Z","UUID":"0d4cfd4e-2104-4436-9761-f7cd68097a4a","Handler":null,"Name":"","Endpoint":""}
Jul 28 22:10:22.296706 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:22.296684    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 22:10:22.296838 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:22.296727    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 22:10:22.401231 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:22.401200    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:22.401392 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:22.401333    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:22.401458 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:22.401388    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:22.401512 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:22.401496    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:22.605902 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:22.605810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28" event={"ID":"6594c0a4-33f6-4633-afbb-3e3e50c81bf2","Type":"ContainerStarted","Data":"7469f91be95ee880fdbbfa3bbabeafdee8a1ae7df122c0bc46c51669d31b2610"}
Jul 28 22:10:23.401335 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:23.401256    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:23.401494 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:23.401396    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:23.610375 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:23.610340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28" event={"ID":"6594c0a4-33f6-4633-afbb-3e3e50c81bf2","Type":"ContainerStarted","Data":"80d3f2a108e5e181be80cf775f2f16689d097618d41b7b9deff4905dd39c3613"}
Jul 28 22:10:23.614025 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:23.614000    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" event={"ID":"1b1aabba-0081-45c1-a027-cdbae9442174","Type":"ContainerStarted","Data":"8e68bfdb38f0ab96c015116a7ae34ff8aa81c22e5eb81325d060b0b676f92f75"}
Jul 28 22:10:23.629398 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:23.629341    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhn28" podStartSLOduration=3.704930734 podStartE2EDuration="24.629320192s" totalImagesPullingTime="20.924389458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:23.628889997 +0000 UTC m=+25.751237920" watchObservedRunningTime="2026-07-28 22:10:23.629320192 +0000 UTC m=+25.751668135"
Jul 28 22:10:24.402001 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:24.401780    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:24.402213 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:24.401780    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:24.402213 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:24.402138    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:24.402213 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:24.402167    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:24.816336 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:24.816307    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:24.870723 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:24.870684    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:25.401381 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:25.401299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:25.401695 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:25.401430    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:25.620802 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:25.620768    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c4023fd-ac12-4f44-ba1d-51693e370007" containerID="9224155bdaf2dc1a151390aa2d5b6661a8426992f1f6534caaa9020117d81af8" exitCode=0
Jul 28 22:10:25.620977 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:25.620850    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlbf4" event={"ID":"8c4023fd-ac12-4f44-ba1d-51693e370007","Type":"ContainerDied","Data":"9224155bdaf2dc1a151390aa2d5b6661a8426992f1f6534caaa9020117d81af8"}
Jul 28 22:10:25.626948 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:25.626919    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" event={"ID":"1b1aabba-0081-45c1-a027-cdbae9442174","Type":"ContainerStarted","Data":"940ae4e0e49bfae85c6e39c028a6fe07621d9614e52a220e0f9b28c250807c68"}
Jul 28 22:10:25.627190 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:25.627170    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:25.627296 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:25.627226    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:25.642612 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:25.642591    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:25.665717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:25.665624    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh" podStartSLOduration=7.822577462 podStartE2EDuration="26.665609637s" totalImagesPullingTime="18.843032175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:25.665240396 +0000 UTC m=+27.787588305" watchObservedRunningTime="2026-07-28 22:10:25.665609637 +0000 UTC m=+27.787957560"
Jul 28 22:10:25.968394 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:25.968187    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:26.067149 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.067115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:26.166830 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.166802    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:26.267948 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.267912    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:26.400746 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.400707    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:26.400746 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.400746    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:26.400985 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:26.400848    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:26.400985 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:26.400930    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:26.629918 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.629842    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:26.645420 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.645388    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:26.698702 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.698672    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:26.796004 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.795969    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:26.849507 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.849479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:26.970253 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:26.970223    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:27.030856 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.030797    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:27.089536 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.089508    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:27.400820 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.400787    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:27.401078 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:27.400911    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:27.437741 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.437715    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4dpq8"]
Jul 28 22:10:27.437893 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.437817    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:27.437945 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:27.437912    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:27.440772 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.440745    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kdlqb"]
Jul 28 22:10:27.441499 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.441471    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-th9pq"]
Jul 28 22:10:27.441600 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.441562    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:27.441679 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:27.441657    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:27.504664 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.504622    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:27.571625 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.571589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:27.633515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.633484    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c4023fd-ac12-4f44-ba1d-51693e370007" containerID="9d3d95c4212405f350f2e6645dbba4b40f90b1f87090f7e374dca3e8ddb9678a" exitCode=0
Jul 28 22:10:27.633679 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.633578    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:27.633679 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.633570    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlbf4" event={"ID":"8c4023fd-ac12-4f44-ba1d-51693e370007","Type":"ContainerDied","Data":"9d3d95c4212405f350f2e6645dbba4b40f90b1f87090f7e374dca3e8ddb9678a"}
Jul 28 22:10:27.633779 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:27.633674    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:27.724471 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:27.724446    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4prbw_ba035fdb-cc39-4bfa-aa6b-81d417c51ad1/dns-node-resolver/0.log"
Jul 28 22:10:28.591325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:28.591290    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:28.642372 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:28.642344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:29.126197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:29.126015    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x75qb_389f391a-1877-421e-932f-2dae44bf6738/node-ca/0.log"
Jul 28 22:10:29.400767 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:29.400676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:29.400941 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:29.400676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:29.400941 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:29.400801    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:29.400941 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:29.400676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:29.400941 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:29.400856    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:29.400941 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:29.400929    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:29.482201 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:29.482169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:29.640443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:29.640414    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c4023fd-ac12-4f44-ba1d-51693e370007" containerID="c2c14d24db5a23b980ff53ff18f9e6aff5e32010ad6a1be5d80cda74f0d90b7d" exitCode=0
Jul 28 22:10:29.640810 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:29.640470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlbf4" event={"ID":"8c4023fd-ac12-4f44-ba1d-51693e370007","Type":"ContainerDied","Data":"c2c14d24db5a23b980ff53ff18f9e6aff5e32010ad6a1be5d80cda74f0d90b7d"}
Jul 28 22:10:31.110803 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:31.110766    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:31.401117 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:31.401075    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:31.401279 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:31.401075    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:31.401279 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:31.401220    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:31.401398 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:31.401299    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:31.401398 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:31.401082    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:31.401507 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:31.401398    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:31.901707 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:31.901640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:31.961529 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:31.961491    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:31.994426 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:31.994394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:31.994578 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:31.994558    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:31.994659 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:31.994633    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret podName:94c1f799-00f0-4a61-b199-8db35aa6169c nodeName:}" failed. No retries permitted until 2026-07-28 22:11:03.99461251 +0000 UTC m=+66.116960386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret") pod "global-pull-secret-syncer-4dpq8" (UID: "94c1f799-00f0-4a61-b199-8db35aa6169c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:10:32.094960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:32.094922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:32.095127 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:32.095069    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:32.095174 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:32.095147    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs podName:28de00f9-d2c8-4535-b6aa-29296c3ae000 nodeName:}" failed. No retries permitted until 2026-07-28 22:11:04.09513069 +0000 UTC m=+66.217478560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs") pod "network-metrics-daemon-kdlqb" (UID: "28de00f9-d2c8-4535-b6aa-29296c3ae000") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:10:32.195390 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:32.195318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:32.195786 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:32.195472    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:10:32.195786 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:32.195492    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:10:32.195786 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:32.195502    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hxbnj for pod openshift-network-diagnostics/network-check-target-th9pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:32.195786 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:32.195550    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj podName:4e8133ea-cd7d-4ee6-9d82-86fc41d0d532 nodeName:}" failed. No retries permitted until 2026-07-28 22:11:04.195536735 +0000 UTC m=+66.317884610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxbnj" (UniqueName: "kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj") pod "network-check-target-th9pq" (UID: "4e8133ea-cd7d-4ee6-9d82-86fc41d0d532") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:10:32.536565 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:32.536532    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:32.759473 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:32.759438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:32.915947 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:32.915864    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:33.016756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:33.016717    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:33.117647 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:33.117601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:33.245065 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:33.245031    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:33.400813 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:33.400764    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:33.400813 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:33.400811    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:33.401057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:33.400879    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:33.401057 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:33.400903    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:33.401057 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:33.400988    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:33.401210 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:33.401079    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:34.032655 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:34.032610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:34.638277 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:34.638246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:34.655907 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:34.655875    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c4023fd-ac12-4f44-ba1d-51693e370007" containerID="c538f6367a697e049bd93ab934ad794166509dfdb2b0df8e07a88a88aa431e57" exitCode=0
Jul 28 22:10:34.656060 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:34.655940    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlbf4" event={"ID":"8c4023fd-ac12-4f44-ba1d-51693e370007","Type":"ContainerDied","Data":"c538f6367a697e049bd93ab934ad794166509dfdb2b0df8e07a88a88aa431e57"}
Jul 28 22:10:34.834522 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:34.834492    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:35.031745 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:35.031704    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:35.235397 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:35.235368    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:35.401411 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:35.401325    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:35.401411 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:35.401395    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:35.401411 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:35.401392    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:35.401610 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:35.401483    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:35.401610 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:35.401594    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:35.401681 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:35.401666    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:35.661223 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:35.661139    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c4023fd-ac12-4f44-ba1d-51693e370007" containerID="2c4fa4b6f4c00bb3c520e221829197d7865ffe4db6b5dae037f34607f856f80d" exitCode=0
Jul 28 22:10:35.661223 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:35.661195    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlbf4" event={"ID":"8c4023fd-ac12-4f44-ba1d-51693e370007","Type":"ContainerDied","Data":"2c4fa4b6f4c00bb3c520e221829197d7865ffe4db6b5dae037f34607f856f80d"}
Jul 28 22:10:36.135767 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:36.135738    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:36.236578 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:36.236540    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:36.293646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:36.293619    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:36.628277 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:36.628240    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:36.667064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:36.667033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlbf4" event={"ID":"8c4023fd-ac12-4f44-ba1d-51693e370007","Type":"ContainerStarted","Data":"427c5a9938101261256de71c299c1e6f3487b6740b5ce32e625cf7eea93e2ab8"}
Jul 28 22:10:36.690169 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:36.690095    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hlbf4" podStartSLOduration=8.021595408 podStartE2EDuration="37.690076709s" totalImagesPullingTime="29.668481301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:36.68955831 +0000 UTC m=+38.811906215" watchObservedRunningTime="2026-07-28 22:10:36.690076709 +0000 UTC m=+38.812424602"
Jul 28 22:10:36.890249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:36.890169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:37.032118 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:37.032058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:37.085987 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:37.085951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:37.306844 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:37.306810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:37.356753 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:37.356718    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:37.401677 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:37.401646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:37.401839 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:37.401735    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:37.401839 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:37.401745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:37.401839 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:37.401741    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:37.401839 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:37.401830    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:37.401974 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:37.401902    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:37.434465 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:37.434430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:38.033949 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:38.033917    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:38.134609 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:38.134572    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:38.237194 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:38.237156    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:38.431888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:38.431811    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:38.817085 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:38.817049    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:38.870978 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:38.870927    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:39.030723 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:39.030689    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:39.400974 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:39.400943    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:39.401478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:39.401060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:39.401478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:39.401090    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:39.401478 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:39.401073    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:39.401478 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:39.401179    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:39.401478 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:39.401263    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:39.426091 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:39.426060    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:39.816390 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:39.816361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:39.869595 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:39.869558    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:40.230815 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:40.230783    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:40.638567 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:40.638492    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:40.834018 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:40.833983    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:41.032280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:41.032246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:41.093413 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:41.093382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:41.239272 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:41.239232    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:41.400900 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:41.400812    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:41.401075 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:41.400917    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:41.401075 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:41.400922    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:41.401075 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:41.401016    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:41.401075 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:41.400922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:41.401323 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:41.401156    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:42.030824 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:42.030788    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:42.430863 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:42.430776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:42.840420 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:42.840384    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:43.033161 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:43.033132    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:43.229977 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:43.229945    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:43.401243 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:43.401215    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:43.401463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:43.401319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:43.401463 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:43.401321    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:43.401463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:43.401323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:43.401463 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:43.401405    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:43.401603 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:43.401511    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:45.401546 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:45.401512    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:45.401971 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:45.401512    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:45.401971 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:45.401637    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:45.401971 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:45.401630    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:45.401971 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:45.401707    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:45.401971 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:45.401764    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:47.401566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:47.401531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:47.401994 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:47.401531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:47.401994 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:47.401647    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:47.401994 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:47.401531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:47.401994 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:47.401740    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:47.401994 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:47.401821    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:49.401446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:49.401408    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:49.401446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:49.401432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:49.401864 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:49.401530    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:49.401864 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:49.401582    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:49.401864 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:49.401670    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:49.401864 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:49.401737    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:50.870027 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:50.869988    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:50.963326 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:50.963295    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:51.062646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:51.062605    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:51.171958 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:51.171868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:51.401033 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:51.401011    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:51.401286 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:51.401011    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:51.401286 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:51.401116    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th9pq" podUID="4e8133ea-cd7d-4ee6-9d82-86fc41d0d532"
Jul 28 22:10:51.401286 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:51.401203    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdlqb" podUID="28de00f9-d2c8-4535-b6aa-29296c3ae000"
Jul 28 22:10:51.401286 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:51.401011    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:51.401286 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:51.401288    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4dpq8" podUID="94c1f799-00f0-4a61-b199-8db35aa6169c"
Jul 28 22:10:51.524892 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:51.524857    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:51.585426 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:51.585390    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:51.638265 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:51.638233    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:52.001484 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.001456    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:52.186587 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.186555    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:52.209308 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.209291    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeReady"
Jul 28 22:10:52.209418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.209411    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 22:10:52.256550 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.256467    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5rhj2"]
Jul 28 22:10:52.263916 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.263882    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:52.277559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.277532    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wlm5v"]
Jul 28 22:10:52.277708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.277657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.280509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.280484    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcmq5\""
Jul 28 22:10:52.280638 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.280511    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 22:10:52.280734 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.280721    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 22:10:52.297527 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.297461    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5rhj2"]
Jul 28 22:10:52.297527 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.297504    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlm5v"]
Jul 28 22:10:52.297678 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.297579    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlm5v"
Jul 28 22:10:52.300368 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.300346    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 22:10:52.300490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.300382    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 22:10:52.300880 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.300866    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lsdr6\""
Jul 28 22:10:52.300928 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.300869    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 22:10:52.340302 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.340270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-tmp-dir\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.340460 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.340318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddjvz\" (UniqueName: \"kubernetes.io/projected/80f95a34-965e-4d1f-907c-142af4243ca8-kube-api-access-ddjvz\") pod \"ingress-canary-wlm5v\" (UID: \"80f95a34-965e-4d1f-907c-142af4243ca8\") " pod="openshift-ingress-canary/ingress-canary-wlm5v"
Jul 28 22:10:52.340460 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.340338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24rlf\" (UniqueName: \"kubernetes.io/projected/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-kube-api-access-24rlf\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.340460 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.340374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-config-volume\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.340460 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.340423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-metrics-tls\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.340584 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.340478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80f95a34-965e-4d1f-907c-142af4243ca8-cert\") pod \"ingress-canary-wlm5v\" (UID: \"80f95a34-965e-4d1f-907c-142af4243ca8\") " pod="openshift-ingress-canary/ingress-canary-wlm5v"
Jul 28 22:10:52.358827 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.358798    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8tzhz"]
Jul 28 22:10:52.379697 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.379662    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8tzhz"]
Jul 28 22:10:52.379841 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.379782    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.380452 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.380423    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:52.382720 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.382700    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 22:10:52.382847 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.382746    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 22:10:52.382847 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.382829    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 22:10:52.382955 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.382868    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 22:10:52.382955 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.382927    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r6q89\""
Jul 28 22:10:52.440755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.440721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddjvz\" (UniqueName: \"kubernetes.io/projected/80f95a34-965e-4d1f-907c-142af4243ca8-kube-api-access-ddjvz\") pod \"ingress-canary-wlm5v\" (UID: \"80f95a34-965e-4d1f-907c-142af4243ca8\") " pod="openshift-ingress-canary/ingress-canary-wlm5v"
Jul 28 22:10:52.440755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.440754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24rlf\" (UniqueName: \"kubernetes.io/projected/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-kube-api-access-24rlf\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.441016 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.440778    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e3f7284-9265-42d4-be4b-a570241a0313-crio-socket\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.441016 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.440808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-config-volume\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.441016 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.440835    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3f7284-9265-42d4-be4b-a570241a0313-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.441016 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.440871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-metrics-tls\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.441016 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.440959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80f95a34-965e-4d1f-907c-142af4243ca8-cert\") pod \"ingress-canary-wlm5v\" (UID: \"80f95a34-965e-4d1f-907c-142af4243ca8\") " pod="openshift-ingress-canary/ingress-canary-wlm5v"
Jul 28 22:10:52.441016 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.440995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3f7284-9265-42d4-be4b-a570241a0313-data-volume\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.441479 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.441029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-tmp-dir\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.441479 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.441052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3f7284-9265-42d4-be4b-a570241a0313-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.441479 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.441127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6r27\" (UniqueName: \"kubernetes.io/projected/2e3f7284-9265-42d4-be4b-a570241a0313-kube-api-access-g6r27\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.441479 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.441366    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-tmp-dir\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.441686 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.441624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-config-volume\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.441686 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.441664    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:52.444981 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.444957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-metrics-tls\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.445067 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.444998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80f95a34-965e-4d1f-907c-142af4243ca8-cert\") pod \"ingress-canary-wlm5v\" (UID: \"80f95a34-965e-4d1f-907c-142af4243ca8\") " pod="openshift-ingress-canary/ingress-canary-wlm5v"
Jul 28 22:10:52.448935 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.448913    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24rlf\" (UniqueName: \"kubernetes.io/projected/752619e7-d2bb-4aa9-a2c9-fdbf24a1b643-kube-api-access-24rlf\") pod \"dns-default-5rhj2\" (UID: \"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643\") " pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.448935 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.448923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddjvz\" (UniqueName: \"kubernetes.io/projected/80f95a34-965e-4d1f-907c-142af4243ca8-kube-api-access-ddjvz\") pod \"ingress-canary-wlm5v\" (UID: \"80f95a34-965e-4d1f-907c-142af4243ca8\") " pod="openshift-ingress-canary/ingress-canary-wlm5v"
Jul 28 22:10:52.505577 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.505547    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:52.541580 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.541508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3f7284-9265-42d4-be4b-a570241a0313-data-volume\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.541580 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.541549    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3f7284-9265-42d4-be4b-a570241a0313-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.541723 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.541586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6r27\" (UniqueName: \"kubernetes.io/projected/2e3f7284-9265-42d4-be4b-a570241a0313-kube-api-access-g6r27\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.541723 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.541616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e3f7284-9265-42d4-be4b-a570241a0313-crio-socket\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.541723 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.541646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3f7284-9265-42d4-be4b-a570241a0313-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.541854 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.541834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e3f7284-9265-42d4-be4b-a570241a0313-crio-socket\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.541955 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.541935    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3f7284-9265-42d4-be4b-a570241a0313-data-volume\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.542200 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.542171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e3f7284-9265-42d4-be4b-a570241a0313-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.547299 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.547274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e3f7284-9265-42d4-be4b-a570241a0313-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.550383 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.550363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6r27\" (UniqueName: \"kubernetes.io/projected/2e3f7284-9265-42d4-be4b-a570241a0313-kube-api-access-g6r27\") pod \"insights-runtime-extractor-8tzhz\" (UID: \"2e3f7284-9265-42d4-be4b-a570241a0313\") " pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.587385 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.587352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:52.605906 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.605877    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlm5v"
Jul 28 22:10:52.606940 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.606914    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:52.689220 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.689184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8tzhz"
Jul 28 22:10:52.809265 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.808249    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlm5v"]
Jul 28 22:10:52.815376 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:52.815305    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f95a34_965e_4d1f_907c_142af4243ca8.slice/crio-8deda8dbf4f194dd0ee4bec56ec51785003147c927989347e378e9365ddc4585 WatchSource:0}: Error finding container 8deda8dbf4f194dd0ee4bec56ec51785003147c927989347e378e9365ddc4585: Status 404 returned error can't find the container with id 8deda8dbf4f194dd0ee4bec56ec51785003147c927989347e378e9365ddc4585
Jul 28 22:10:52.819685 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.819661    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5rhj2"]
Jul 28 22:10:52.822468 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:52.822445    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752619e7_d2bb_4aa9_a2c9_fdbf24a1b643.slice/crio-9dcb786ab7830b024ba62973b5be747144439f584bcc46ccd79672bf9357eb21 WatchSource:0}: Error finding container 9dcb786ab7830b024ba62973b5be747144439f584bcc46ccd79672bf9357eb21: Status 404 returned error can't find the container with id 9dcb786ab7830b024ba62973b5be747144439f584bcc46ccd79672bf9357eb21
Jul 28 22:10:52.876629 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:52.876599    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8tzhz"]
Jul 28 22:10:52.879906 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:10:52.879883    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3f7284_9265_42d4_be4b_a570241a0313.slice/crio-54173a780a41e9c94a4d4dae4ecd3eee42578204ad3a15508ab6550b3d173db2 WatchSource:0}: Error finding container 54173a780a41e9c94a4d4dae4ecd3eee42578204ad3a15508ab6550b3d173db2: Status 404 returned error can't find the container with id 54173a780a41e9c94a4d4dae4ecd3eee42578204ad3a15508ab6550b3d173db2
Jul 28 22:10:53.394464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.394425    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:53.400948 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.400909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:10:53.400948 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.400924    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:10:53.401325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.401305    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:10:53.404819 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.404796    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 22:10:53.404919 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.404799    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q7z5n\""
Jul 28 22:10:53.404919 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.404896    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 22:10:53.405296 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.405278    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 22:10:53.405383 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.405339    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7dq7d\""
Jul 28 22:10:53.405436 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.405381    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 22:10:53.600691 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.600637    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:53.711255 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.711059    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8tzhz" event={"ID":"2e3f7284-9265-42d4-be4b-a570241a0313","Type":"ContainerStarted","Data":"54173a780a41e9c94a4d4dae4ecd3eee42578204ad3a15508ab6550b3d173db2"}
Jul 28 22:10:53.712652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.712578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlm5v" event={"ID":"80f95a34-965e-4d1f-907c-142af4243ca8","Type":"ContainerStarted","Data":"8deda8dbf4f194dd0ee4bec56ec51785003147c927989347e378e9365ddc4585"}
Jul 28 22:10:53.714024 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:53.713973    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rhj2" event={"ID":"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643","Type":"ContainerStarted","Data":"9dcb786ab7830b024ba62973b5be747144439f584bcc46ccd79672bf9357eb21"}
Jul 28 22:10:54.016753 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:54.016617    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:54.214340 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:54.214276    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:54.269915 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:54.269697    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:54.812818 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:54.812769    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:54.912878 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:54.912844    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:55.013386 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:55.013349    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:55.065079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:55.064938    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:57.344661 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.344625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:57.449534 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.449505    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:57.512131 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.512081    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:57.514996 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.514971    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"]
Jul 28 22:10:57.533979 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.533953    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"]
Jul 28 22:10:57.534153 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.534060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"
Jul 28 22:10:57.536690 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.536669    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 22:10:57.536834 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.536746    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7sxql\""
Jul 28 22:10:57.582499 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.582467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a40d15d5-0327-4557-90b6-0179ffa1a0cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-ckk4w\" (UID: \"a40d15d5-0327-4557-90b6-0179ffa1a0cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"
Jul 28 22:10:57.684061 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.683965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a40d15d5-0327-4557-90b6-0179ffa1a0cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-ckk4w\" (UID: \"a40d15d5-0327-4557-90b6-0179ffa1a0cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"
Jul 28 22:10:57.684243 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:57.684161    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 22:10:57.684303 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:10:57.684255    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40d15d5-0327-4557-90b6-0179ffa1a0cd-tls-certificates podName:a40d15d5-0327-4557-90b6-0179ffa1a0cd nodeName:}" failed. No retries permitted until 2026-07-28 22:10:58.184231859 +0000 UTC m=+60.306579730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a40d15d5-0327-4557-90b6-0179ffa1a0cd-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-ckk4w" (UID: "a40d15d5-0327-4557-90b6-0179ffa1a0cd") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 22:10:57.856403 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.856370    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:57.994277 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:57.994243    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:58.097026 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.096985    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:58.187567 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.187533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a40d15d5-0327-4557-90b6-0179ffa1a0cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-ckk4w\" (UID: \"a40d15d5-0327-4557-90b6-0179ffa1a0cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"
Jul 28 22:10:58.190095 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.190058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a40d15d5-0327-4557-90b6-0179ffa1a0cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-ckk4w\" (UID: \"a40d15d5-0327-4557-90b6-0179ffa1a0cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"
Jul 28 22:10:58.194646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.194616    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:58.293634 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.293559    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:58.447964 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.447929    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7sxql\""
Jul 28 22:10:58.455175 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.455151    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"
Jul 28 22:10:58.646851 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.646811    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xphqh"
Jul 28 22:10:58.764145 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.763903    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"]
Jul 28 22:10:58.933237 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.933160    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:58.982743 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:58.982712    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:59.335573 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.335510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:59.393221 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.393184    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:10:59.734195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.734153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8tzhz" event={"ID":"2e3f7284-9265-42d4-be4b-a570241a0313","Type":"ContainerStarted","Data":"4be0754113b74f07eb8473b91e39fb12bf5c80b3b7b1d9be0aec6c09f3fe2510"}
Jul 28 22:10:59.736264 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.736233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rhj2" event={"ID":"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643","Type":"ContainerStarted","Data":"cb1c8c061add98b4d7783280bcc6ffce95b8ef983273d2585824b0314db72e1a"}
Jul 28 22:10:59.736415 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.736288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rhj2" event={"ID":"752619e7-d2bb-4aa9-a2c9-fdbf24a1b643","Type":"ContainerStarted","Data":"6ee3c29d1654da7faa4e2ca1b80b4fa47a13f175991a898fec21c6f1d72ea289"}
Jul 28 22:10:59.736415 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.736320    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:10:59.737765 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.737735    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w" event={"ID":"a40d15d5-0327-4557-90b6-0179ffa1a0cd","Type":"ContainerStarted","Data":"4b6f2e5323a244e5ad1e4d1c0460967f423237bb6936c0e288a6c9363ad0905d"}
Jul 28 22:10:59.739338 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.739315    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlm5v" event={"ID":"80f95a34-965e-4d1f-907c-142af4243ca8","Type":"ContainerStarted","Data":"fcb213430776f8cc56910cbedcff942eeff4308b2b5a96f0a6d18ab2c1988cbe"}
Jul 28 22:10:59.754606 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.754562    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5rhj2" podStartSLOduration=1.958316852 podStartE2EDuration="7.754546328s" totalImagesPullingTime="5.796229476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:59.753191829 +0000 UTC m=+61.875539750" watchObservedRunningTime="2026-07-28 22:10:59.754546328 +0000 UTC m=+61.876894221"
Jul 28 22:10:59.768887 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:10:59.768835    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wlm5v" podStartSLOduration=1.962236646 podStartE2EDuration="7.768817202s" totalImagesPullingTime="5.806580556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:59.768287122 +0000 UTC m=+61.890635030" watchObservedRunningTime="2026-07-28 22:10:59.768817202 +0000 UTC m=+61.891165096"
Jul 28 22:11:00.743877 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:00.743823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w" event={"ID":"a40d15d5-0327-4557-90b6-0179ffa1a0cd","Type":"ContainerStarted","Data":"e9731faeab6b0cea0501f8fd7b54e7fb27e27016946a9c3bc061a7fc8e57387e"}
Jul 28 22:11:00.744325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:00.744201    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"
Jul 28 22:11:00.749466 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:00.749442    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w"
Jul 28 22:11:00.761059 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:00.760853    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ckk4w" podStartSLOduration=2.566300333 podStartE2EDuration="3.760840677s" totalImagesPullingTime="1.194540344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:00.760439426 +0000 UTC m=+62.882787340" watchObservedRunningTime="2026-07-28 22:11:00.760840677 +0000 UTC m=+62.883188549"
Jul 28 22:11:01.539228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.539188    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:01.576233 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.576202    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-c22pg"]
Jul 28 22:11:01.579893 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.579868    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.582309 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.582276    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 22:11:01.582452 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.582427    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 22:11:01.582753 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.582730    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 22:11:01.582851 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.582761    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 22:11:01.583652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.583627    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4pbvt\""
Jul 28 22:11:01.583747 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.583713    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 22:11:01.589913 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.589833    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-c22pg"]
Jul 28 22:11:01.610976 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.610949    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:01.613828 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.613786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.613965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.613867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.613965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.613902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.613965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.613936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xmk\" (UniqueName: \"kubernetes.io/projected/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-kube-api-access-t8xmk\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.633347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.633316    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b8f64896c-rcv87"]
Jul 28 22:11:01.637910 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.637892    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.641064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.640614    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 22:11:01.641064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.640648    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 22:11:01.641064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.640756    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 22:11:01.641064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.640942    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 22:11:01.641064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.640978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jr9lx\""
Jul 28 22:11:01.641064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.641012    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 22:11:01.641418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.641155    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 22:11:01.641418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.641242    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 22:11:01.647038 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.646973    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8f64896c-rcv87"]
Jul 28 22:11:01.714391 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.714581 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714424    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xmk\" (UniqueName: \"kubernetes.io/projected/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-kube-api-access-t8xmk\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.714581 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-oauth-serving-cert\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.714581 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.714581 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-serving-cert\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.714581 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-service-ca\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.714581 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714526    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-oauth-config\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.714887 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:11:01.714620    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 22:11:01.714887 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.714887 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-config\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.714887 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.714697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4xg\" (UniqueName: \"kubernetes.io/projected/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-kube-api-access-dm4xg\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.714887 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:11:01.714755    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-prometheus-operator-tls podName:6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f nodeName:}" failed. No retries permitted until 2026-07-28 22:11:02.214728331 +0000 UTC m=+64.337076218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-c22pg" (UID: "6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f") : secret "prometheus-operator-tls" not found
Jul 28 22:11:01.715161 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.715141    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.717211 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.717182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.723847 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.723821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xmk\" (UniqueName: \"kubernetes.io/projected/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-kube-api-access-t8xmk\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:01.815215 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.815130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-oauth-serving-cert\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.815215 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.815191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-serving-cert\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.815215 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.815208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-service-ca\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.815775 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.815228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-oauth-config\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.815775 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.815295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-config\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.815775 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.815336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4xg\" (UniqueName: \"kubernetes.io/projected/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-kube-api-access-dm4xg\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.815908 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.815884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-oauth-serving-cert\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.815963 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.815944    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-service-ca\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.816124 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.816075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-config\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.817669 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.817650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-serving-cert\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.817883 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.817862    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-oauth-config\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.823900 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.823879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4xg\" (UniqueName: \"kubernetes.io/projected/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-kube-api-access-dm4xg\") pod \"console-6b8f64896c-rcv87\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") " pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:01.949446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:01.949407    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:02.180652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:02.180619    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:02.218375 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:02.218352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:02.220800 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:02.220774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-c22pg\" (UID: \"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:02.232058 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:02.232030    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8f64896c-rcv87"]
Jul 28 22:11:02.234741 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:02.234717    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3aa494_6eab_48ae_98d7_3f1a7abee30c.slice/crio-527271f330a18f921b7665e6f30916329966a291a2845ecab2121811fe927dce WatchSource:0}: Error finding container 527271f330a18f921b7665e6f30916329966a291a2845ecab2121811fe927dce: Status 404 returned error can't find the container with id 527271f330a18f921b7665e6f30916329966a291a2845ecab2121811fe927dce
Jul 28 22:11:02.490460 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:02.490427    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg"
Jul 28 22:11:02.615166 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:02.615125    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-c22pg"]
Jul 28 22:11:02.618782 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:02.618758    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba2fadb_e0b8_4ddc_9fe0_f6fb1862be7f.slice/crio-607de5b235a321d3345ad01f52809434b884e117c34e99e3a2bf5102927cbbf9 WatchSource:0}: Error finding container 607de5b235a321d3345ad01f52809434b884e117c34e99e3a2bf5102927cbbf9: Status 404 returned error can't find the container with id 607de5b235a321d3345ad01f52809434b884e117c34e99e3a2bf5102927cbbf9
Jul 28 22:11:02.753672 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:02.753589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8tzhz" event={"ID":"2e3f7284-9265-42d4-be4b-a570241a0313","Type":"ContainerStarted","Data":"4b56b3ad26329f3e2ff3939f6a21d5077caa0f29700426cbb85a8bb3d3509292"}
Jul 28 22:11:02.755002 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:02.754971    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg" event={"ID":"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f","Type":"ContainerStarted","Data":"607de5b235a321d3345ad01f52809434b884e117c34e99e3a2bf5102927cbbf9"}
Jul 28 22:11:02.756342 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:02.756312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8f64896c-rcv87" event={"ID":"ed3aa494-6eab-48ae-98d7-3f1a7abee30c","Type":"ContainerStarted","Data":"527271f330a18f921b7665e6f30916329966a291a2845ecab2121811fe927dce"}
Jul 28 22:11:03.056144 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:03.056030    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:03.138757 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:03.138681    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:04.035909 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.035769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:11:04.038786 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.038752    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 22:11:04.049459 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.049349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94c1f799-00f0-4a61-b199-8db35aa6169c-original-pull-secret\") pod \"global-pull-secret-syncer-4dpq8\" (UID: \"94c1f799-00f0-4a61-b199-8db35aa6169c\") " pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:11:04.136923 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.136839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:11:04.139702 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.139675    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 22:11:04.153327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.153299    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28de00f9-d2c8-4535-b6aa-29296c3ae000-metrics-certs\") pod \"network-metrics-daemon-kdlqb\" (UID: \"28de00f9-d2c8-4535-b6aa-29296c3ae000\") " pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:11:04.216970 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.216938    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q7z5n\""
Jul 28 22:11:04.225328 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.225296    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdlqb"
Jul 28 22:11:04.226529 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.226493    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4dpq8"
Jul 28 22:11:04.238166 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.238127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:11:04.240914 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.240888    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 22:11:04.251315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.251288    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 22:11:04.262233 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.262208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbnj\" (UniqueName: \"kubernetes.io/projected/4e8133ea-cd7d-4ee6-9d82-86fc41d0d532-kube-api-access-hxbnj\") pod \"network-check-target-th9pq\" (UID: \"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532\") " pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:11:04.536631 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.536598    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7dq7d\""
Jul 28 22:11:04.544950 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:04.544922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:11:05.216658 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:05.216618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:05.275073 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:05.275039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:06.612137 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:06.612085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:06.691523 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:06.691485    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:07.990640 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:07.990527    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kdlqb"]
Jul 28 22:11:07.997206 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:07.997172    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28de00f9_d2c8_4535_b6aa_29296c3ae000.slice/crio-8f8c941a52178b93f3b32a30fc56801ffef0ab145f6b8b53bc07a8dc988dd487 WatchSource:0}: Error finding container 8f8c941a52178b93f3b32a30fc56801ffef0ab145f6b8b53bc07a8dc988dd487: Status 404 returned error can't find the container with id 8f8c941a52178b93f3b32a30fc56801ffef0ab145f6b8b53bc07a8dc988dd487
Jul 28 22:11:08.196011 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.195919    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:08.216248 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.216210    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4dpq8"]
Jul 28 22:11:08.219396 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.219370    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-th9pq"]
Jul 28 22:11:08.220241 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:08.220214    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c1f799_00f0_4a61_b199_8db35aa6169c.slice/crio-a2e2986be3799e8ed702bb5a25ca7dfcb90f6ae94713cb6dc29eda9db048bf48 WatchSource:0}: Error finding container a2e2986be3799e8ed702bb5a25ca7dfcb90f6ae94713cb6dc29eda9db048bf48: Status 404 returned error can't find the container with id a2e2986be3799e8ed702bb5a25ca7dfcb90f6ae94713cb6dc29eda9db048bf48
Jul 28 22:11:08.222484 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:08.222462    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8133ea_cd7d_4ee6_9d82_86fc41d0d532.slice/crio-59b39da62b41f76cbfd629f8f7c0caea34ecfe538fbf58dea020fbe7c842464d WatchSource:0}: Error finding container 59b39da62b41f76cbfd629f8f7c0caea34ecfe538fbf58dea020fbe7c842464d: Status 404 returned error can't find the container with id 59b39da62b41f76cbfd629f8f7c0caea34ecfe538fbf58dea020fbe7c842464d
Jul 28 22:11:08.781580 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.781545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8tzhz" event={"ID":"2e3f7284-9265-42d4-be4b-a570241a0313","Type":"ContainerStarted","Data":"937335e3cee4e9b5c5438107bbd4fe6a47ae23f2cc3237cf36bce54f462e1e10"}
Jul 28 22:11:08.782669 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.782644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-th9pq" event={"ID":"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532","Type":"ContainerStarted","Data":"59b39da62b41f76cbfd629f8f7c0caea34ecfe538fbf58dea020fbe7c842464d"}
Jul 28 22:11:08.784208 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.784179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg" event={"ID":"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f","Type":"ContainerStarted","Data":"4b69d5e9fe5f3e23ab8c96063cfa9e1597146d9e395949a147f60ba1451cf0bd"}
Jul 28 22:11:08.784304 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.784234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg" event={"ID":"6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f","Type":"ContainerStarted","Data":"d0949e884c2a8ba377cb543303a4fb49d91ebf7a7aed762a4c4d1e0364fd32ac"}
Jul 28 22:11:08.785721 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.785693    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8f64896c-rcv87" event={"ID":"ed3aa494-6eab-48ae-98d7-3f1a7abee30c","Type":"ContainerStarted","Data":"0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b"}
Jul 28 22:11:08.786738 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.786718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4dpq8" event={"ID":"94c1f799-00f0-4a61-b199-8db35aa6169c","Type":"ContainerStarted","Data":"a2e2986be3799e8ed702bb5a25ca7dfcb90f6ae94713cb6dc29eda9db048bf48"}
Jul 28 22:11:08.787673 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.787651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdlqb" event={"ID":"28de00f9-d2c8-4535-b6aa-29296c3ae000","Type":"ContainerStarted","Data":"8f8c941a52178b93f3b32a30fc56801ffef0ab145f6b8b53bc07a8dc988dd487"}
Jul 28 22:11:08.798547 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.798504    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8tzhz" podStartSLOduration=1.982177415 podStartE2EDuration="16.798490712s" totalImagesPullingTime="14.816313297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:10:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:08.798066262 +0000 UTC m=+70.920414155" watchObservedRunningTime="2026-07-28 22:11:08.798490712 +0000 UTC m=+70.920838595"
Jul 28 22:11:08.815463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.815365    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b8f64896c-rcv87" podStartSLOduration=2.215903048 podStartE2EDuration="7.815348336s" totalImagesPullingTime="5.599445288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:08.814503371 +0000 UTC m=+70.936851263" watchObservedRunningTime="2026-07-28 22:11:08.815348336 +0000 UTC m=+70.937696235"
Jul 28 22:11:08.832509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.832462    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-c22pg" podStartSLOduration=2.617377374 podStartE2EDuration="7.83244114s" totalImagesPullingTime="5.215063766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:08.831311321 +0000 UTC m=+70.953659215" watchObservedRunningTime="2026-07-28 22:11:08.83244114 +0000 UTC m=+70.954789034"
Jul 28 22:11:08.983175 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:08.983139    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:09.094277 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.094074    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:09.610864 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.610828    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-799757d59d-rvfjr"]
Jul 28 22:11:09.613077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.613055    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.624145 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.623671    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 22:11:09.628413 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.627240    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799757d59d-rvfjr"]
Jul 28 22:11:09.682701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.682657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-service-ca\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.682888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.682715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-oauth-serving-cert\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.682888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.682745    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-console-config\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.682888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.682774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-serving-cert\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.682888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.682813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-trusted-ca-bundle\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.682888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.682846    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmlbr\" (UniqueName: \"kubernetes.io/projected/c4165e03-01f3-4449-ac4b-b8140148c143-kube-api-access-fmlbr\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.683171 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.682894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-oauth-config\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.746787 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.746748    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5rhj2"
Jul 28 22:11:09.783455 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.783423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-service-ca\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.783455 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.783460    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-oauth-serving-cert\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.783684 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.783476    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-console-config\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.783684 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.783535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-serving-cert\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.783684 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.783589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-trusted-ca-bundle\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.783684 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.783616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmlbr\" (UniqueName: \"kubernetes.io/projected/c4165e03-01f3-4449-ac4b-b8140148c143-kube-api-access-fmlbr\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.783944 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.783710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-oauth-config\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.784259 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.784234    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-oauth-serving-cert\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.784379 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.784276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-console-config\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.784559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.784534    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-trusted-ca-bundle\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.784774 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.784756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-service-ca\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.787012 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.786993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-serving-cert\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.787283 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.787262    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-oauth-config\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.792596 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.792569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmlbr\" (UniqueName: \"kubernetes.io/projected/c4165e03-01f3-4449-ac4b-b8140148c143-kube-api-access-fmlbr\") pod \"console-799757d59d-rvfjr\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") " pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.796086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.796060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdlqb" event={"ID":"28de00f9-d2c8-4535-b6aa-29296c3ae000","Type":"ContainerStarted","Data":"d7b300fa02be53c728f2e83b87ccaf3491180fe8292c3e6cfd06d029e2e2f8d5"}
Jul 28 22:11:09.796203 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.796160    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdlqb" event={"ID":"28de00f9-d2c8-4535-b6aa-29296c3ae000","Type":"ContainerStarted","Data":"45edbdc385f4c9b91031d5caf82f35f3d5243c171f7b89f4e1d489bce4cc0733"}
Jul 28 22:11:09.819306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.819215    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kdlqb" podStartSLOduration=70.002995279 podStartE2EDuration="1m10.819193499s" totalImagesPullingTime="816.19822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:09.818490678 +0000 UTC m=+71.940838574" watchObservedRunningTime="2026-07-28 22:11:09.819193499 +0000 UTC m=+71.941541392"
Jul 28 22:11:09.834451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.834416    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:09.930949 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.930680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:09.935599 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:09.935575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:10.035049 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:10.034910    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:10.083568 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:10.083537    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799757d59d-rvfjr"]
Jul 28 22:11:10.086916 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:10.086886    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4165e03_01f3_4449_ac4b_b8140148c143.slice/crio-2fe93766ee9c0a905041809be17cbfc8f11af1b3d0cfcd5344867b9d40021813 WatchSource:0}: Error finding container 2fe93766ee9c0a905041809be17cbfc8f11af1b3d0cfcd5344867b9d40021813: Status 404 returned error can't find the container with id 2fe93766ee9c0a905041809be17cbfc8f11af1b3d0cfcd5344867b9d40021813
Jul 28 22:11:10.676053 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:10.676012    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:10.802738 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:10.802696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799757d59d-rvfjr" event={"ID":"c4165e03-01f3-4449-ac4b-b8140148c143","Type":"ContainerStarted","Data":"26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8"}
Jul 28 22:11:10.802919 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:10.802764    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799757d59d-rvfjr" event={"ID":"c4165e03-01f3-4449-ac4b-b8140148c143","Type":"ContainerStarted","Data":"2fe93766ee9c0a905041809be17cbfc8f11af1b3d0cfcd5344867b9d40021813"}
Jul 28 22:11:10.823187 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:10.823128    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-799757d59d-rvfjr" podStartSLOduration=1.823056805 podStartE2EDuration="1.823056805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:10.821964976 +0000 UTC m=+72.944312888" watchObservedRunningTime="2026-07-28 22:11:10.823056805 +0000 UTC m=+72.945404699"
Jul 28 22:11:10.961220 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:10.961131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:10.998682 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:10.998645    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"]
Jul 28 22:11:11.002461 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.002432    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"]
Jul 28 22:11:11.002628 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.002567    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.004976 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.004948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.005093 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.005054    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 22:11:11.005093 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.005069    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dnq98\""
Jul 28 22:11:11.005246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.005057    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 22:11:11.007801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.007761    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 22:11:11.008184 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.008164    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 22:11:11.008184 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.008180    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nf2sl\""
Jul 28 22:11:11.008485 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.008469    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 22:11:11.012043 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.012023    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rkmgt"]
Jul 28 22:11:11.014607 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.014574    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"]
Jul 28 22:11:11.014715 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.014614    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.018509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.017780    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"]
Jul 28 22:11:11.018509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.018041    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 22:11:11.018509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.018064    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 22:11:11.018509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.018305    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 22:11:11.018509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.018507    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n8cxc\""
Jul 28 22:11:11.046038 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.046003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:11.094998 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.094964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.095165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.095165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3847606-a998-4e33-977d-981a3181029a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.095165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a87fbe1-e0c9-421d-a157-b29d988cc06d-metrics-client-ca\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.095383 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnbjh\" (UniqueName: \"kubernetes.io/projected/5a87fbe1-e0c9-421d-a157-b29d988cc06d-kube-api-access-pnbjh\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.095383 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a87fbe1-e0c9-421d-a157-b29d988cc06d-root\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.095383 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3847606-a998-4e33-977d-981a3181029a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.095383 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-tls\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.095383 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a87fbe1-e0c9-421d-a157-b29d988cc06d-sys\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.095383 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.095645 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-accelerators-collector-config\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.095645 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.095645 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.095645 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-textfile\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.095645 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095548    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-wtmp\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.095645 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095627    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9tl\" (UniqueName: \"kubernetes.io/projected/a3847606-a998-4e33-977d-981a3181029a-kube-api-access-bc9tl\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.095865 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.095865 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tsx9\" (UniqueName: \"kubernetes.io/projected/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-api-access-7tsx9\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.095865 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.095727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3847606-a998-4e33-977d-981a3181029a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.108347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.108314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:11.196391 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9tl\" (UniqueName: \"kubernetes.io/projected/a3847606-a998-4e33-977d-981a3181029a-kube-api-access-bc9tl\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.196570 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.196570 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tsx9\" (UniqueName: \"kubernetes.io/projected/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-api-access-7tsx9\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.196570 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196476    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3847606-a998-4e33-977d-981a3181029a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.196570 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196515    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.196570 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.196570 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3847606-a998-4e33-977d-981a3181029a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a87fbe1-e0c9-421d-a157-b29d988cc06d-metrics-client-ca\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnbjh\" (UniqueName: \"kubernetes.io/projected/5a87fbe1-e0c9-421d-a157-b29d988cc06d-kube-api-access-pnbjh\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a87fbe1-e0c9-421d-a157-b29d988cc06d-root\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3847606-a998-4e33-977d-981a3181029a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-tls\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a87fbe1-e0c9-421d-a157-b29d988cc06d-sys\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-accelerators-collector-config\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a87fbe1-e0c9-421d-a157-b29d988cc06d-root\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.196888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.196839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.197440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.197532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a87fbe1-e0c9-421d-a157-b29d988cc06d-sys\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.197680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3847606-a998-4e33-977d-981a3181029a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.197724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.197759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-textfile\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.197782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.197789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-wtmp\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.197896    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-wtmp\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.198074    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-accelerators-collector-config\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.198358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-textfile\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.198376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a87fbe1-e0c9-421d-a157-b29d988cc06d-metrics-client-ca\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.198700 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.198663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.199909 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.199860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.200391 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.200344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.200600 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.200578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-tls\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.200742 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.200713    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a87fbe1-e0c9-421d-a157-b29d988cc06d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.200852 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.200834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3847606-a998-4e33-977d-981a3181029a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.201936 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.201914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3847606-a998-4e33-977d-981a3181029a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.206120 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.206082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tsx9\" (UniqueName: \"kubernetes.io/projected/f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3-kube-api-access-7tsx9\") pod \"kube-state-metrics-56d9f944b4-5sbml\" (UID: \"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.206273 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.206254    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnbjh\" (UniqueName: \"kubernetes.io/projected/5a87fbe1-e0c9-421d-a157-b29d988cc06d-kube-api-access-pnbjh\") pod \"node-exporter-rkmgt\" (UID: \"5a87fbe1-e0c9-421d-a157-b29d988cc06d\") " pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.206387 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.206363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9tl\" (UniqueName: \"kubernetes.io/projected/a3847606-a998-4e33-977d-981a3181029a-kube-api-access-bc9tl\") pod \"openshift-state-metrics-b8f49f5c6-zvr7l\" (UID: \"a3847606-a998-4e33-977d-981a3181029a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.208285 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.208235    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:11.309437 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.309381    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:11.316578 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.316556    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"
Jul 28 22:11:11.326717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.326658    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"
Jul 28 22:11:11.335421 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.335399    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rkmgt"
Jul 28 22:11:11.408779 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.408729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:11.909889 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.909849    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:11.949703 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.949662    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:11.949922 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.949796    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:11.956120 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.956058    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:11.990708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.990648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:11.995322 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.995299    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 22:11:11.998118 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:11.998073    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.001079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.000726    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 22:11:12.001079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.000787    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 22:11:12.001079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.000831    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ctvxd\""
Jul 28 22:11:12.001079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.000864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 22:11:12.001079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.000877    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 22:11:12.001079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.000953    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 22:11:12.001079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.000983    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 22:11:12.001542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.001095    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 22:11:12.001542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.001218    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 22:11:12.001542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.001452    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 22:11:12.015381 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.015359    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 22:11:12.105459 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.105459 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.105754 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.105754 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105530    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-config-out\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.105754 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.105754 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.105754 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-web-config\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.106005 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105775    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.106005 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.106005 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.106005 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105885    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhzl\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-kube-api-access-wrhzl\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.106005 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.106005 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.105940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207282 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-web-config\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207282 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207282 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhzl\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-kube-api-access-wrhzl\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:11:12.207443    2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:11:12.207519    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-main-tls podName:6b2e445b-db40-4c23-9cc4-51497416156e nodeName:}" failed. No retries permitted until 2026-07-28 22:11:12.707499037 +0000 UTC m=+74.829846917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e") : secret "alertmanager-main-tls" not found
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207543    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.207572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-config-out\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.208044 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.208044 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.207663    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.208724 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.208428    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.210899 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.210528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.210899 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.210685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.210899 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.210533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.211247 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.211223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.211414 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.211379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-config-out\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.211517 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.211420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.211908 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.211889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.212002 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.211953    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.212002 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.211953    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.212217 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.212196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-web-config\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.215780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.215762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhzl\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-kube-api-access-wrhzl\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.311213 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.311168    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:12.713816 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.713771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.716469 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.716438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.815069 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.815041    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:12.903426 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.903390    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76fd97f949-lxgh7"]
Jul 28 22:11:12.906230 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.906206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:12.908937 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.908907    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 22:11:12.908937 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.908920    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 22:11:12.909124 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.908980    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a260gu3ueh6ug\""
Jul 28 22:11:12.909284 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.909264    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 22:11:12.909284 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.909279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:11:12.909443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.909320    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 22:11:12.909443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.909396    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b969d\""
Jul 28 22:11:12.909748 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.909581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 22:11:12.918734 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:12.918711    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76fd97f949-lxgh7"]
Jul 28 22:11:12.932713 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:12.932675    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a87fbe1_e0c9_421d_a157_b29d988cc06d.slice/crio-06b1ffd0e6c6160bc8b69dd16209dba60e706c0f5ce1b59376c9182730d12ace WatchSource:0}: Error finding container 06b1ffd0e6c6160bc8b69dd16209dba60e706c0f5ce1b59376c9182730d12ace: Status 404 returned error can't find the container with id 06b1ffd0e6c6160bc8b69dd16209dba60e706c0f5ce1b59376c9182730d12ace
Jul 28 22:11:13.029870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.028872    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.029870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.028929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-tls\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.029870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.028972    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-grpc-tls\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.029870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.029008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.029870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.029034    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx7vm\" (UniqueName: \"kubernetes.io/projected/9bf7316a-83ed-486a-a247-9445f637e721-kube-api-access-mx7vm\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.029870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.029073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.029870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.029123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.029870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.029165    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7316a-83ed-486a-a247-9445f637e721-metrics-client-ca\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.109981 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.109956    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml"]
Jul 28 22:11:13.112170 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:13.112130    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61e6d68_6b06_4f2d_b1c1_7f44bc4a5af3.slice/crio-41aa7ae858791e449f2b55e1099b2fd4f4260dbf2745106f3ba3100a067943ee WatchSource:0}: Error finding container 41aa7ae858791e449f2b55e1099b2fd4f4260dbf2745106f3ba3100a067943ee: Status 404 returned error can't find the container with id 41aa7ae858791e449f2b55e1099b2fd4f4260dbf2745106f3ba3100a067943ee
Jul 28 22:11:13.129668 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.129637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7316a-83ed-486a-a247-9445f637e721-metrics-client-ca\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.129762 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.129686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.129762 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.129716    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-tls\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.129762 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.129749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-grpc-tls\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.129909 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.129773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.129909 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.129789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx7vm\" (UniqueName: \"kubernetes.io/projected/9bf7316a-83ed-486a-a247-9445f637e721-kube-api-access-mx7vm\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.130376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.130322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.130475 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.130371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.130548 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.130488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7316a-83ed-486a-a247-9445f637e721-metrics-client-ca\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.132568 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.132533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-tls\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.132887 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.132862    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.133058 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.133041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.133286 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.133269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-grpc-tls\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.133500 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.133476    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.133645 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.133630    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bf7316a-83ed-486a-a247-9445f637e721-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.140514 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.140492    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx7vm\" (UniqueName: \"kubernetes.io/projected/9bf7316a-83ed-486a-a247-9445f637e721-kube-api-access-mx7vm\") pod \"thanos-querier-76fd97f949-lxgh7\" (UID: \"9bf7316a-83ed-486a-a247-9445f637e721\") " pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.220193 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.220091    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:13.333894 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.333869    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l"]
Jul 28 22:11:13.336046 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:13.336014    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3847606_a998_4e33_977d_981a3181029a.slice/crio-7ed075dbdb78b419f14f5a3b5c0c3bd59be1d7cd0fac7f9502736b2eeb4f47db WatchSource:0}: Error finding container 7ed075dbdb78b419f14f5a3b5c0c3bd59be1d7cd0fac7f9502736b2eeb4f47db: Status 404 returned error can't find the container with id 7ed075dbdb78b419f14f5a3b5c0c3bd59be1d7cd0fac7f9502736b2eeb4f47db
Jul 28 22:11:13.342887 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.342849    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 22:11:13.347066 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:13.347042    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2e445b_db40_4c23_9cc4_51497416156e.slice/crio-1bebb3fcec97d82096934faec3916637e4435e2c051a3fd656501e8a2ae14748 WatchSource:0}: Error finding container 1bebb3fcec97d82096934faec3916637e4435e2c051a3fd656501e8a2ae14748: Status 404 returned error can't find the container with id 1bebb3fcec97d82096934faec3916637e4435e2c051a3fd656501e8a2ae14748
Jul 28 22:11:13.363623 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.363599    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76fd97f949-lxgh7"]
Jul 28 22:11:13.366154 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:13.366127    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf7316a_83ed_486a_a247_9445f637e721.slice/crio-caa1752f0f66417d7e34f4e2316af045be274bcae41b2055c438a2a20b30ad99 WatchSource:0}: Error finding container caa1752f0f66417d7e34f4e2316af045be274bcae41b2055c438a2a20b30ad99: Status 404 returned error can't find the container with id caa1752f0f66417d7e34f4e2316af045be274bcae41b2055c438a2a20b30ad99
Jul 28 22:11:13.817456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.817415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerStarted","Data":"1bebb3fcec97d82096934faec3916637e4435e2c051a3fd656501e8a2ae14748"}
Jul 28 22:11:13.821195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.821164    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l" event={"ID":"a3847606-a998-4e33-977d-981a3181029a","Type":"ContainerStarted","Data":"d571640e43708150e04288a2b451211db93574f068633d8473ffa0095078dcc6"}
Jul 28 22:11:13.821327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.821229    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l" event={"ID":"a3847606-a998-4e33-977d-981a3181029a","Type":"ContainerStarted","Data":"d1d03f18df3f08297b35027dc8146c9130f47a6b02e562dd7932254d7fe393db"}
Jul 28 22:11:13.821327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.821260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l" event={"ID":"a3847606-a998-4e33-977d-981a3181029a","Type":"ContainerStarted","Data":"7ed075dbdb78b419f14f5a3b5c0c3bd59be1d7cd0fac7f9502736b2eeb4f47db"}
Jul 28 22:11:13.823725 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.823650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7" event={"ID":"9bf7316a-83ed-486a-a247-9445f637e721","Type":"ContainerStarted","Data":"caa1752f0f66417d7e34f4e2316af045be274bcae41b2055c438a2a20b30ad99"}
Jul 28 22:11:13.825507 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.825483    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rkmgt" event={"ID":"5a87fbe1-e0c9-421d-a157-b29d988cc06d","Type":"ContainerStarted","Data":"06b1ffd0e6c6160bc8b69dd16209dba60e706c0f5ce1b59376c9182730d12ace"}
Jul 28 22:11:13.827828 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.827775    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-th9pq" event={"ID":"4e8133ea-cd7d-4ee6-9d82-86fc41d0d532","Type":"ContainerStarted","Data":"0716c2d7093261e8f7b7d1ceb7247804276eb10de76db122478058ad6b0981d9"}
Jul 28 22:11:13.827946 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.827925    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:11:13.830652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.830602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4dpq8" event={"ID":"94c1f799-00f0-4a61-b199-8db35aa6169c","Type":"ContainerStarted","Data":"f59567767dfce22b53023bcecced5b198832610a1e1669017aa8f60e35980e54"}
Jul 28 22:11:13.833826 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.833759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml" event={"ID":"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3","Type":"ContainerStarted","Data":"41aa7ae858791e449f2b55e1099b2fd4f4260dbf2745106f3ba3100a067943ee"}
Jul 28 22:11:13.839348 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.839271    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:13.844580 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.844155    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-th9pq" podStartSLOduration=70.133690413 podStartE2EDuration="1m14.844136612s" totalImagesPullingTime="4.710446199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:13.843806934 +0000 UTC m=+75.966154862" watchObservedRunningTime="2026-07-28 22:11:13.844136612 +0000 UTC m=+75.966484517"
Jul 28 22:11:13.859804 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.859753    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4dpq8" podStartSLOduration=70.136961614 podStartE2EDuration="1m14.859738069s" totalImagesPullingTime="4.722776455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:13.8582955 +0000 UTC m=+75.980643419" watchObservedRunningTime="2026-07-28 22:11:13.859738069 +0000 UTC m=+75.982085940"
Jul 28 22:11:13.939930 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:13.939881    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:14.070460 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:14.070376    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:14.172887 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:14.171717    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:14.840148 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:14.839869    2560 generic.go:356] "Generic (PLEG): container finished" podID="5a87fbe1-e0c9-421d-a157-b29d988cc06d" containerID="425e40bdaabb7c616641b2d4516db9ec2d3e0e31679fcbc4b2c2d9ae55cdf845" exitCode=0
Jul 28 22:11:14.840148 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:14.839986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rkmgt" event={"ID":"5a87fbe1-e0c9-421d-a157-b29d988cc06d","Type":"ContainerDied","Data":"425e40bdaabb7c616641b2d4516db9ec2d3e0e31679fcbc4b2c2d9ae55cdf845"}
Jul 28 22:11:15.238082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.237909    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8577ff485f-gxjgx"]
Jul 28 22:11:15.241621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.241560    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.244370 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.244338    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 22:11:15.244370 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.244358    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 22:11:15.244559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.244374    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pn52w\""
Jul 28 22:11:15.244559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.244344    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 22:11:15.244775 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.244758    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 22:11:15.244835 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.244760    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8eevusm1bs5vk\""
Jul 28 22:11:15.251608 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.251521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8577ff485f-gxjgx"]
Jul 28 22:11:15.351440 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.351404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bz6h\" (UniqueName: \"kubernetes.io/projected/93e993a2-bcb4-4ed4-8d2e-55898a337031-kube-api-access-8bz6h\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.351621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.351492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e993a2-bcb4-4ed4-8d2e-55898a337031-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.351621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.351524    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/93e993a2-bcb4-4ed4-8d2e-55898a337031-client-ca-bundle\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.351621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.351578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/93e993a2-bcb4-4ed4-8d2e-55898a337031-metrics-server-audit-profiles\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.351772 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.351621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/93e993a2-bcb4-4ed4-8d2e-55898a337031-secret-metrics-server-client-certs\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.351772 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.351660    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/93e993a2-bcb4-4ed4-8d2e-55898a337031-audit-log\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.351772 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.351703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/93e993a2-bcb4-4ed4-8d2e-55898a337031-secret-metrics-server-tls\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.453756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.453670    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/93e993a2-bcb4-4ed4-8d2e-55898a337031-secret-metrics-server-tls\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.453756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.453720    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bz6h\" (UniqueName: \"kubernetes.io/projected/93e993a2-bcb4-4ed4-8d2e-55898a337031-kube-api-access-8bz6h\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.453984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.453791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e993a2-bcb4-4ed4-8d2e-55898a337031-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.453984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.453814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/93e993a2-bcb4-4ed4-8d2e-55898a337031-client-ca-bundle\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.453984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.453842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/93e993a2-bcb4-4ed4-8d2e-55898a337031-metrics-server-audit-profiles\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.453984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.453885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/93e993a2-bcb4-4ed4-8d2e-55898a337031-secret-metrics-server-client-certs\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.453984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.453915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/93e993a2-bcb4-4ed4-8d2e-55898a337031-audit-log\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.454360 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.454329    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/93e993a2-bcb4-4ed4-8d2e-55898a337031-audit-log\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.455451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.455393    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/93e993a2-bcb4-4ed4-8d2e-55898a337031-metrics-server-audit-profiles\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.455595 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.455570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e993a2-bcb4-4ed4-8d2e-55898a337031-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.456921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.456902    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/93e993a2-bcb4-4ed4-8d2e-55898a337031-client-ca-bundle\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.457061 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.457036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/93e993a2-bcb4-4ed4-8d2e-55898a337031-secret-metrics-server-tls\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.457253 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.457200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/93e993a2-bcb4-4ed4-8d2e-55898a337031-secret-metrics-server-client-certs\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.462549 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.462523    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bz6h\" (UniqueName: \"kubernetes.io/projected/93e993a2-bcb4-4ed4-8d2e-55898a337031-kube-api-access-8bz6h\") pod \"metrics-server-8577ff485f-gxjgx\" (UID: \"93e993a2-bcb4-4ed4-8d2e-55898a337031\") " pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.554521 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.554439    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:15.691514 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.691461    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:15.712211 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.711764    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"]
Jul 28 22:11:15.714329 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.714303    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:15.716837 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.716793    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 22:11:15.717004 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.716863    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vfctg\""
Jul 28 22:11:15.721937 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.721878    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"]
Jul 28 22:11:15.750834 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.750776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:15.825893 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.825783    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799757d59d-rvfjr"]
Jul 28 22:11:15.847038 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.846952    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rkmgt" event={"ID":"5a87fbe1-e0c9-421d-a157-b29d988cc06d","Type":"ContainerStarted","Data":"9d2a09b2930d69b0a4bb39e8b9599ea9c3429bad199edab007e91dc13692fcad"}
Jul 28 22:11:15.847038 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.847000    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rkmgt" event={"ID":"5a87fbe1-e0c9-421d-a157-b29d988cc06d","Type":"ContainerStarted","Data":"54b0dc39b9ffb98ad5e29af6011da320f02600b649adafd54995f64d0fa8289e"}
Jul 28 22:11:15.853672 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.853641    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml" event={"ID":"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3","Type":"ContainerStarted","Data":"00aa320e0ecbf665694a0b6bf121d4e0a45619d551ed6c9042d55f8b710dec71"}
Jul 28 22:11:15.853782 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.853705    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml" event={"ID":"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3","Type":"ContainerStarted","Data":"c38f013118a3e812fe026227e599844aa33f71ab7bc6b68de0c3144b05f2097e"}
Jul 28 22:11:15.853782 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.853736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml" event={"ID":"f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3","Type":"ContainerStarted","Data":"10c7b1907909f9f7993718536dd044aba4223e8e1395741bd0b921d5cba47161"}
Jul 28 22:11:15.859708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.859399    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xszqh\" (UniqueName: \"kubernetes.io/projected/2eba8e1b-809c-4c88-9248-79ff922c0312-kube-api-access-xszqh\") pod \"monitoring-plugin-68b7fc9574-nj8k8\" (UID: \"2eba8e1b-809c-4c88-9248-79ff922c0312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:15.859708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.859406    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ff897fdc-55mn8"]
Jul 28 22:11:15.859708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.859518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2eba8e1b-809c-4c88-9248-79ff922c0312-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-nj8k8\" (UID: \"2eba8e1b-809c-4c88-9248-79ff922c0312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:15.863813 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.863781    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b2e445b-db40-4c23-9cc4-51497416156e" containerID="b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f" exitCode=0
Jul 28 22:11:15.866041 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.865912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerDied","Data":"b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f"}
Jul 28 22:11:15.866123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.866042    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:15.866919 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.866870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l" event={"ID":"a3847606-a998-4e33-977d-981a3181029a","Type":"ContainerStarted","Data":"785683b06a383807cacea493c0b31df5aa2dedc526936a0583e11e4978890860"}
Jul 28 22:11:15.869722 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.869700    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff897fdc-55mn8"]
Jul 28 22:11:15.894744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.894706    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:15.905733 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.905687    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rkmgt" podStartSLOduration=4.749162886 podStartE2EDuration="5.905670078s" totalImagesPullingTime="1.076229172s" totalInitContainerRuntime="80.27802ms" isStatefulPod=true podCreationTimestamp="2026-07-28 22:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:15.874733713 +0000 UTC m=+77.997081628" watchObservedRunningTime="2026-07-28 22:11:15.905670078 +0000 UTC m=+78.028017950"
Jul 28 22:11:15.921858 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.921830    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8577ff485f-gxjgx"]
Jul 28 22:11:15.923890 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.923801    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5sbml" podStartSLOduration=4.084581691 podStartE2EDuration="5.923779749s" totalImagesPullingTime="1.839198058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:15.922561673 +0000 UTC m=+78.044909595" watchObservedRunningTime="2026-07-28 22:11:15.923779749 +0000 UTC m=+78.046127644"
Jul 28 22:11:15.942947 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.942896    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zvr7l" podStartSLOduration=4.407044351 podStartE2EDuration="5.94287584s" totalImagesPullingTime="1.535831489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:15.942623815 +0000 UTC m=+78.064971711" watchObservedRunningTime="2026-07-28 22:11:15.94287584 +0000 UTC m=+78.065223740"
Jul 28 22:11:15.960829 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.960687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-oauth-config\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:15.960829 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.960724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8244\" (UniqueName: \"kubernetes.io/projected/7949852f-3125-40ca-b556-39a355019725-kube-api-access-l8244\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:15.960829 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.960782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2eba8e1b-809c-4c88-9248-79ff922c0312-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-nj8k8\" (UID: \"2eba8e1b-809c-4c88-9248-79ff922c0312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:15.960829 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.960830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-trusted-ca-bundle\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:15.961126 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.961068    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-serving-cert\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:15.961196 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.961179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xszqh\" (UniqueName: \"kubernetes.io/projected/2eba8e1b-809c-4c88-9248-79ff922c0312-kube-api-access-xszqh\") pod \"monitoring-plugin-68b7fc9574-nj8k8\" (UID: \"2eba8e1b-809c-4c88-9248-79ff922c0312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:15.962228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.962181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-console-config\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:15.962331 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.962309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-service-ca\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:15.962433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.962414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-oauth-serving-cert\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:15.964295 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.964276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2eba8e1b-809c-4c88-9248-79ff922c0312-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-nj8k8\" (UID: \"2eba8e1b-809c-4c88-9248-79ff922c0312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:15.977993 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:15.977952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xszqh\" (UniqueName: \"kubernetes.io/projected/2eba8e1b-809c-4c88-9248-79ff922c0312-kube-api-access-xszqh\") pod \"monitoring-plugin-68b7fc9574-nj8k8\" (UID: \"2eba8e1b-809c-4c88-9248-79ff922c0312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:16.027169 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.027120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:16.052857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.052824    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:16.062991 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.062959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-oauth-config\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.063159 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.063004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8244\" (UniqueName: \"kubernetes.io/projected/7949852f-3125-40ca-b556-39a355019725-kube-api-access-l8244\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.063159 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.063118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-trusted-ca-bundle\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.063262 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.063209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-serving-cert\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.063304 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.063281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-console-config\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.063349 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.063324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-service-ca\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.063402 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.063353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-oauth-serving-cert\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.064125 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.064080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-oauth-serving-cert\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.064341 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.064285    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-console-config\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.064451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.064322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-trusted-ca-bundle\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.064638 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.064616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-service-ca\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.065795 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.065778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-oauth-config\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.065911 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.065894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-serving-cert\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.073208 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.073153    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8244\" (UniqueName: \"kubernetes.io/projected/7949852f-3125-40ca-b556-39a355019725-kube-api-access-l8244\") pod \"console-ff897fdc-55mn8\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") " pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.150615 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.150582    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:16.155288 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.155257    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-694bf69f78-8rflg"]
Jul 28 22:11:16.159931 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.159911    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"]
Jul 28 22:11:16.160036 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.160017    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.160750 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:16.160727    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eba8e1b_809c_4c88_9248_79ff922c0312.slice/crio-27b9948daab258723ea372944a388ec4fade3ac55dd79e05c9c8bb9cca1fcdb8 WatchSource:0}: Error finding container 27b9948daab258723ea372944a388ec4fade3ac55dd79e05c9c8bb9cca1fcdb8: Status 404 returned error can't find the container with id 27b9948daab258723ea372944a388ec4fade3ac55dd79e05c9c8bb9cca1fcdb8
Jul 28 22:11:16.162814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.162655    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 22:11:16.162814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.162665    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 22:11:16.162814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.162669    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 22:11:16.162814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.162676    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 22:11:16.162814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.162715    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cg78l\""
Jul 28 22:11:16.162814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.162714    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 22:11:16.168566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.168547    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 22:11:16.174677 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.174655    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-694bf69f78-8rflg"]
Jul 28 22:11:16.191434 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.191409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:16.209562 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.209535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:16.264729 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.264695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-secret-telemeter-client\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.265123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.264738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.265123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.264765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070c4254-3b07-487c-a50f-c8b605e20f65-serving-certs-ca-bundle\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.265123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.264784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-telemeter-client-tls\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.265123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.264827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070c4254-3b07-487c-a50f-c8b605e20f65-telemeter-trusted-ca-bundle\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.265123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.264858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070c4254-3b07-487c-a50f-c8b605e20f65-metrics-client-ca\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.265123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.264880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-federate-client-tls\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.265123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.264950    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjlc\" (UniqueName: \"kubernetes.io/projected/070c4254-3b07-487c-a50f-c8b605e20f65-kube-api-access-swjlc\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.317978 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.317947    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff897fdc-55mn8"]
Jul 28 22:11:16.321293 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:16.321265    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7949852f_3125_40ca_b556_39a355019725.slice/crio-8126a039ae3331c1f780ff5ba90b5c46d9f69cdae772f6661c03db5aa2fdb066 WatchSource:0}: Error finding container 8126a039ae3331c1f780ff5ba90b5c46d9f69cdae772f6661c03db5aa2fdb066: Status 404 returned error can't find the container with id 8126a039ae3331c1f780ff5ba90b5c46d9f69cdae772f6661c03db5aa2fdb066
Jul 28 22:11:16.365436 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.365408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.365566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.365455    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070c4254-3b07-487c-a50f-c8b605e20f65-serving-certs-ca-bundle\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.365566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.365482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-telemeter-client-tls\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.365566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.365519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070c4254-3b07-487c-a50f-c8b605e20f65-telemeter-trusted-ca-bundle\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.365566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.365561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070c4254-3b07-487c-a50f-c8b605e20f65-metrics-client-ca\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.365755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.365594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-federate-client-tls\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.365755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.365644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjlc\" (UniqueName: \"kubernetes.io/projected/070c4254-3b07-487c-a50f-c8b605e20f65-kube-api-access-swjlc\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.365755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.365701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-secret-telemeter-client\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.366328 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.366298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070c4254-3b07-487c-a50f-c8b605e20f65-serving-certs-ca-bundle\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.366431 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.366385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070c4254-3b07-487c-a50f-c8b605e20f65-metrics-client-ca\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.366559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.366528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070c4254-3b07-487c-a50f-c8b605e20f65-telemeter-trusted-ca-bundle\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.368515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.368493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-telemeter-client-tls\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.368727 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.368702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-secret-telemeter-client\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.368800 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.368785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.369306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.369291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/070c4254-3b07-487c-a50f-c8b605e20f65-federate-client-tls\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.374030 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.374000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjlc\" (UniqueName: \"kubernetes.io/projected/070c4254-3b07-487c-a50f-c8b605e20f65-kube-api-access-swjlc\") pod \"telemeter-client-694bf69f78-8rflg\" (UID: \"070c4254-3b07-487c-a50f-c8b605e20f65\") " pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.470470 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.470430    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg"
Jul 28 22:11:16.638467 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.638334    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-694bf69f78-8rflg"]
Jul 28 22:11:16.643315 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:16.643281    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070c4254_3b07_487c_a50f_c8b605e20f65.slice/crio-9a92642134bc136d9fe983cc1a7f662747fbf42c4982f66b96ce71a094525c5e WatchSource:0}: Error finding container 9a92642134bc136d9fe983cc1a7f662747fbf42c4982f66b96ce71a094525c5e: Status 404 returned error can't find the container with id 9a92642134bc136d9fe983cc1a7f662747fbf42c4982f66b96ce71a094525c5e
Jul 28 22:11:16.874754 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.874702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg" event={"ID":"070c4254-3b07-487c-a50f-c8b605e20f65","Type":"ContainerStarted","Data":"9a92642134bc136d9fe983cc1a7f662747fbf42c4982f66b96ce71a094525c5e"}
Jul 28 22:11:16.876792 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.876733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx" event={"ID":"93e993a2-bcb4-4ed4-8d2e-55898a337031","Type":"ContainerStarted","Data":"2b985dfbc445abf2a0fc733237f27c521aad09135d8841b5165ef2ef3cc8134f"}
Jul 28 22:11:16.882347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.882227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff897fdc-55mn8" event={"ID":"7949852f-3125-40ca-b556-39a355019725","Type":"ContainerStarted","Data":"7908b3c0f25ec7b57c3dd4ee32afa89ae0c398c4e9678301bc4085619619cfd6"}
Jul 28 22:11:16.882347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.882303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff897fdc-55mn8" event={"ID":"7949852f-3125-40ca-b556-39a355019725","Type":"ContainerStarted","Data":"8126a039ae3331c1f780ff5ba90b5c46d9f69cdae772f6661c03db5aa2fdb066"}
Jul 28 22:11:16.887156 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.887128    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7" event={"ID":"9bf7316a-83ed-486a-a247-9445f637e721","Type":"ContainerStarted","Data":"ecff3c913126ff86421b85aaf696c4f09d7a3a518e2b3f8f9bf74230ab4d6d20"}
Jul 28 22:11:16.887274 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.887205    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7" event={"ID":"9bf7316a-83ed-486a-a247-9445f637e721","Type":"ContainerStarted","Data":"0c79e415caf4939596b01740e6fb83cb3dfb3d32b14d4f33cd42481c746a4f0b"}
Jul 28 22:11:16.887274 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.887246    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7" event={"ID":"9bf7316a-83ed-486a-a247-9445f637e721","Type":"ContainerStarted","Data":"c989ebc9d58fb1b3cffc2244fe8275f7c2a87c2d4dd8edb4125ab4d0abf57357"}
Jul 28 22:11:16.890038 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.889272    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8" event={"ID":"2eba8e1b-809c-4c88-9248-79ff922c0312","Type":"ContainerStarted","Data":"27b9948daab258723ea372944a388ec4fade3ac55dd79e05c9c8bb9cca1fcdb8"}
Jul 28 22:11:16.899682 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:16.899294    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ff897fdc-55mn8" podStartSLOduration=1.899272531 podStartE2EDuration="1.899272531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:16.89870219 +0000 UTC m=+79.021050109" watchObservedRunningTime="2026-07-28 22:11:16.899272531 +0000 UTC m=+79.021620425"
Jul 28 22:11:17.037455 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.037403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:17.108494 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.108452    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:17.210591 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.210451    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:17.211284 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.210885    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 22:11:17.217231 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.215699    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.219490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.218576    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 22:11:17.219490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.218721    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 22:11:17.219490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.218973    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 22:11:17.219490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.218987    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6lj5v7fcke3v1\""
Jul 28 22:11:17.219490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.219231    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 22:11:17.219490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.219263    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 22:11:17.219490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.219457    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 22:11:17.219874 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.219462    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5s22m\""
Jul 28 22:11:17.222133 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.220638    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 22:11:17.222133 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.220841    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 22:11:17.222133 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.221239    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 22:11:17.222133 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.221429    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 22:11:17.222376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.222148    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 22:11:17.222433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.222398    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 22:11:17.225195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.225175    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 22:11:17.234397 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.234319    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 22:11:17.237527 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.237500    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 22:11:17.377438 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377529    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377552    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377577    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmhp\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-kube-api-access-skmhp\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377743    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-config\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.377898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377885    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.379301 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.379301 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377949    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.379301 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.377998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479237    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmhp\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-kube-api-access-skmhp\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479333    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-config\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479458    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479486    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479669    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479695    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.479801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.480292 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.479840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.481262 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.480469    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.481262 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.480569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.481262 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.480779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.481262 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.481116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.486652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.485941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.486652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.486091    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.488052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.487750    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.489763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.488950    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.489763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.489459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.490483 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.490144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.491974 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.491942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.492643 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.492597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.493601 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.493297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.494078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.494010    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-config\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.494342 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.494274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.494685 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.494637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.495210 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.495030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.495210 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.495172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.495558 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.495532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.498428 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.498400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmhp\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-kube-api-access-skmhp\") pod \"prometheus-k8s-0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.554876 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.554840    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:17.897123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.897000    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7" event={"ID":"9bf7316a-83ed-486a-a247-9445f637e721","Type":"ContainerStarted","Data":"e0948f75e24aa995f24b5811d4b40f7a06112b5cb8568aec0fe1a2e0890b2b54"}
Jul 28 22:11:17.897296 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.897146    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7" event={"ID":"9bf7316a-83ed-486a-a247-9445f637e721","Type":"ContainerStarted","Data":"928f1fc66ed6f7abba3ffafd2bdec25f5239763eca2635e6dcc1240d5a13e131"}
Jul 28 22:11:17.897296 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.897185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7" event={"ID":"9bf7316a-83ed-486a-a247-9445f637e721","Type":"ContainerStarted","Data":"0aff77a94293a9dbb4a33acd749fc9b66dd267edf97b44ef7d4c7ae0a00d5305"}
Jul 28 22:11:17.919987 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:17.919920    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7" podStartSLOduration=2.561111006 podStartE2EDuration="5.919895959s" totalImagesPullingTime="3.358784953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:17.917843939 +0000 UTC m=+80.040191874" watchObservedRunningTime="2026-07-28 22:11:17.919895959 +0000 UTC m=+80.042243859"
Jul 28 22:11:18.004568 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:18.004533    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:18.067708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:18.067669    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:18.227367 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:18.227303    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:18.284184 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:18.284060    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:18.902028 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:18.901992    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:19.112129 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.111980    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 22:11:19.115312 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:11:19.115267    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b3e424_3036_48bf_9076_a1316c8312a0.slice/crio-8327d07a88f562f860340fd2b8a7de0cbcb7fbfa7e2ca4f3236a9d07e8c926dd WatchSource:0}: Error finding container 8327d07a88f562f860340fd2b8a7de0cbcb7fbfa7e2ca4f3236a9d07e8c926dd: Status 404 returned error can't find the container with id 8327d07a88f562f860340fd2b8a7de0cbcb7fbfa7e2ca4f3236a9d07e8c926dd
Jul 28 22:11:19.228296 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.228262    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:19.282418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.282386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:19.909585 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.909528    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg" event={"ID":"070c4254-3b07-487c-a50f-c8b605e20f65","Type":"ContainerStarted","Data":"d90400037a9285290f61126aaadf7c28e3d52beceaaa443fd40a5fc498e82f6d"}
Jul 28 22:11:19.910079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.909598    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg" event={"ID":"070c4254-3b07-487c-a50f-c8b605e20f65","Type":"ContainerStarted","Data":"56c4ea3f605ee049a2d4c0169558f2565cd9276dbb2b27357d1b55eadbdbb353"}
Jul 28 22:11:19.910079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.909628    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg" event={"ID":"070c4254-3b07-487c-a50f-c8b605e20f65","Type":"ContainerStarted","Data":"1b2f92cf7f2dff389e2773cd1171d3b37688a213c9a0c622b065490529eab297"}
Jul 28 22:11:19.912020 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.911995    2560 generic.go:356] "Generic (PLEG): container finished" podID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerID="606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6" exitCode=0
Jul 28 22:11:19.912157 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.912082    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerDied","Data":"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"}
Jul 28 22:11:19.912234 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.912189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerStarted","Data":"8327d07a88f562f860340fd2b8a7de0cbcb7fbfa7e2ca4f3236a9d07e8c926dd"}
Jul 28 22:11:19.913822 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.913769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx" event={"ID":"93e993a2-bcb4-4ed4-8d2e-55898a337031","Type":"ContainerStarted","Data":"2386dffd1d4daaef673e38dc5791bd2f6d08eb7b51c7ba9435eed405fdf73f53"}
Jul 28 22:11:19.918944 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.918921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerStarted","Data":"5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a"}
Jul 28 22:11:19.919056 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.918990    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerStarted","Data":"155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878"}
Jul 28 22:11:19.919056 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.919046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerStarted","Data":"d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225"}
Jul 28 22:11:19.919184 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.919082    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerStarted","Data":"f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e"}
Jul 28 22:11:19.919184 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.919135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerStarted","Data":"41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393"}
Jul 28 22:11:19.919184 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.919161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerStarted","Data":"32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6"}
Jul 28 22:11:19.921052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.921020    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8" event={"ID":"2eba8e1b-809c-4c88-9248-79ff922c0312","Type":"ContainerStarted","Data":"b98ebc841a449059cf39a341bc21383b94e6bca07d0ed0b7a84a6472de51fb77"}
Jul 28 22:11:19.922619 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.922538    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:19.926631 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.926610    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8"
Jul 28 22:11:19.931036 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.930978    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:19.931199 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.931082    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-694bf69f78-8rflg" podStartSLOduration=1.5983372340000002 podStartE2EDuration="3.931063925s" totalImagesPullingTime="2.332726691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:19.928171106 +0000 UTC m=+82.050519019" watchObservedRunningTime="2026-07-28 22:11:19.931063925 +0000 UTC m=+82.053411820"
Jul 28 22:11:19.945354 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.945305    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx" podStartSLOduration=1.926635616 podStartE2EDuration="4.945290749s" totalImagesPullingTime="3.018655133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:19.943630305 +0000 UTC m=+82.065978218" watchObservedRunningTime="2026-07-28 22:11:19.945290749 +0000 UTC m=+82.067638619"
Jul 28 22:11:19.970733 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.970679    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.26117559 podStartE2EDuration="8.970662891s" totalImagesPullingTime="4.685328741s" totalInitContainerRuntime="24.15856ms" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:19.969829246 +0000 UTC m=+82.092177158" watchObservedRunningTime="2026-07-28 22:11:19.970662891 +0000 UTC m=+82.093010783"
Jul 28 22:11:19.987723 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:19.987666    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-nj8k8" podStartSLOduration=2.178702624 podStartE2EDuration="4.987646141s" totalImagesPullingTime="2.808943517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:19.985649578 +0000 UTC m=+82.107997495" watchObservedRunningTime="2026-07-28 22:11:19.987646141 +0000 UTC m=+82.109994015"
Jul 28 22:11:20.120418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:20.120274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:20.226930 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:20.226894    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:20.533116 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:20.532992    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:20.633682 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:20.633553    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:20.785912 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:20.785828    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff897fdc-55mn8"]
Jul 28 22:11:23.957735 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:23.957697    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerStarted","Data":"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"}
Jul 28 22:11:23.958121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:23.957800    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerStarted","Data":"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"}
Jul 28 22:11:23.958121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:23.957836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerStarted","Data":"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"}
Jul 28 22:11:23.958121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:23.957861    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerStarted","Data":"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"}
Jul 28 22:11:23.958121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:23.957892    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerStarted","Data":"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"}
Jul 28 22:11:23.958121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:23.957916    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerStarted","Data":"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"}
Jul 28 22:11:24.010598 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:24.010522    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.808141119 podStartE2EDuration="7.010499143s" totalImagesPullingTime="3.173227638s" totalInitContainerRuntime="29.130386ms" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:24.009161977 +0000 UTC m=+86.131509890" watchObservedRunningTime="2026-07-28 22:11:24.010499143 +0000 UTC m=+86.132847037"
Jul 28 22:11:24.877668 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:24.877635    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:24.928173 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:24.928145    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76fd97f949-lxgh7"
Jul 28 22:11:24.955572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:24.955535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:26.192005 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:26.191969    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:27.005152 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:27.005116    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:27.103354 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:27.103319    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:27.556015 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:27.555985    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:11:29.554475 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:29.554435    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:29.611197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:29.611161    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:32.091175 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:32.091144    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8f64896c-rcv87"]
Jul 28 22:11:32.187753 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:32.187728    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:32.287037 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:32.287001    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:32.387671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:32.387589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:35.555441 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:35.555411    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:35.555818 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:35.555471    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:39.808995 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:39.808954    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:39.857877 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:39.857847    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:40.851999 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:40.851962    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-799757d59d-rvfjr" podUID="c4165e03-01f3-4449-ac4b-b8140148c143" containerName="console" containerID="cri-o://26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8" gracePeriod=15
Jul 28 22:11:41.632450 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.632418    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:41.722312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722276    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-trusted-ca-bundle\") pod \"c4165e03-01f3-4449-ac4b-b8140148c143\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") "
Jul 28 22:11:41.722495 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722323    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmlbr\" (UniqueName: \"kubernetes.io/projected/c4165e03-01f3-4449-ac4b-b8140148c143-kube-api-access-fmlbr\") pod \"c4165e03-01f3-4449-ac4b-b8140148c143\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") "
Jul 28 22:11:41.722495 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722367    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-serving-cert\") pod \"c4165e03-01f3-4449-ac4b-b8140148c143\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") "
Jul 28 22:11:41.722495 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722395    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-service-ca\") pod \"c4165e03-01f3-4449-ac4b-b8140148c143\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") "
Jul 28 22:11:41.722495 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722445    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-console-config\") pod \"c4165e03-01f3-4449-ac4b-b8140148c143\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") "
Jul 28 22:11:41.722740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722501    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-oauth-config\") pod \"c4165e03-01f3-4449-ac4b-b8140148c143\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") "
Jul 28 22:11:41.722740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722526    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-oauth-serving-cert\") pod \"c4165e03-01f3-4449-ac4b-b8140148c143\" (UID: \"c4165e03-01f3-4449-ac4b-b8140148c143\") "
Jul 28 22:11:41.722740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722698    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c4165e03-01f3-4449-ac4b-b8140148c143" (UID: "c4165e03-01f3-4449-ac4b-b8140148c143"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:41.722889 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722817    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-console-config" (OuterVolumeSpecName: "console-config") pod "c4165e03-01f3-4449-ac4b-b8140148c143" (UID: "c4165e03-01f3-4449-ac4b-b8140148c143"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:41.722930 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722915    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-trusted-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:41.722995 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.722939    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-console-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:41.723203 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.723051    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c4165e03-01f3-4449-ac4b-b8140148c143" (UID: "c4165e03-01f3-4449-ac4b-b8140148c143"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:41.723296 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.723268    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-service-ca" (OuterVolumeSpecName: "service-ca") pod "c4165e03-01f3-4449-ac4b-b8140148c143" (UID: "c4165e03-01f3-4449-ac4b-b8140148c143"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:41.724862 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.724830    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c4165e03-01f3-4449-ac4b-b8140148c143" (UID: "c4165e03-01f3-4449-ac4b-b8140148c143"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:11:41.724952 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.724906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c4165e03-01f3-4449-ac4b-b8140148c143" (UID: "c4165e03-01f3-4449-ac4b-b8140148c143"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:11:41.724952 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.724908    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4165e03-01f3-4449-ac4b-b8140148c143-kube-api-access-fmlbr" (OuterVolumeSpecName: "kube-api-access-fmlbr") pod "c4165e03-01f3-4449-ac4b-b8140148c143" (UID: "c4165e03-01f3-4449-ac4b-b8140148c143"). InnerVolumeSpecName "kube-api-access-fmlbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:11:41.823662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.823619    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmlbr\" (UniqueName: \"kubernetes.io/projected/c4165e03-01f3-4449-ac4b-b8140148c143-kube-api-access-fmlbr\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:41.823662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.823653    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-serving-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:41.823662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.823667    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:41.823897 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.823678    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4165e03-01f3-4449-ac4b-b8140148c143-console-oauth-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:41.823897 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:41.823698    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4165e03-01f3-4449-ac4b-b8140148c143-oauth-serving-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:42.043392 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.043298    2560 generic.go:356] "Generic (PLEG): container finished" podID="c4165e03-01f3-4449-ac4b-b8140148c143" containerID="26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8" exitCode=0
Jul 28 22:11:42.043392 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.043361    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799757d59d-rvfjr"
Jul 28 22:11:42.043392 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.043383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799757d59d-rvfjr" event={"ID":"c4165e03-01f3-4449-ac4b-b8140148c143","Type":"ContainerDied","Data":"26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8"}
Jul 28 22:11:42.043923 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.043453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799757d59d-rvfjr" event={"ID":"c4165e03-01f3-4449-ac4b-b8140148c143","Type":"ContainerDied","Data":"2fe93766ee9c0a905041809be17cbfc8f11af1b3d0cfcd5344867b9d40021813"}
Jul 28 22:11:42.043923 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.043474    2560 scope.go:117] "RemoveContainer" containerID="26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8"
Jul 28 22:11:42.052449 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.052434    2560 scope.go:117] "RemoveContainer" containerID="26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8"
Jul 28 22:11:42.052671 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:11:42.052654    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8\": container with ID starting with 26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8 not found: ID does not exist" containerID="26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8"
Jul 28 22:11:42.052715 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.052681    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8"} err="failed to get container status \"26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8\": rpc error: code = NotFound desc = could not find container \"26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8\": container with ID starting with 26d8c86ef147fe174691a13e599a0965651300cee5da997c1b5586d02b8de0b8 not found: ID does not exist"
Jul 28 22:11:42.064812 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.064750    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799757d59d-rvfjr"]
Jul 28 22:11:42.066629 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.066608    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-799757d59d-rvfjr"]
Jul 28 22:11:42.406502 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:42.406410    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4165e03-01f3-4449-ac4b-b8140148c143" path="/var/lib/kubelet/pods/c4165e03-01f3-4449-ac4b-b8140148c143/volumes"
Jul 28 22:11:43.349664 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:43.349628    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:43.451396 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:43.451364    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:44.842736 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:44.842708    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-th9pq"
Jul 28 22:11:45.807935 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:45.807892    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ff897fdc-55mn8" podUID="7949852f-3125-40ca-b556-39a355019725" containerName="console" containerID="cri-o://7908b3c0f25ec7b57c3dd4ee32afa89ae0c398c4e9678301bc4085619619cfd6" gracePeriod=15
Jul 28 22:11:46.061519 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.061457    2560 generic.go:356] "Generic (PLEG): container finished" podID="7949852f-3125-40ca-b556-39a355019725" containerID="7908b3c0f25ec7b57c3dd4ee32afa89ae0c398c4e9678301bc4085619619cfd6" exitCode=0
Jul 28 22:11:46.061822 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.061534    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff897fdc-55mn8" event={"ID":"7949852f-3125-40ca-b556-39a355019725","Type":"ContainerDied","Data":"7908b3c0f25ec7b57c3dd4ee32afa89ae0c398c4e9678301bc4085619619cfd6"}
Jul 28 22:11:46.061822 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.061579    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff897fdc-55mn8" event={"ID":"7949852f-3125-40ca-b556-39a355019725","Type":"ContainerDied","Data":"8126a039ae3331c1f780ff5ba90b5c46d9f69cdae772f6661c03db5aa2fdb066"}
Jul 28 22:11:46.061822 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.061596    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8126a039ae3331c1f780ff5ba90b5c46d9f69cdae772f6661c03db5aa2fdb066"
Jul 28 22:11:46.069972 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.069952    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:46.158980 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.158953    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-service-ca\") pod \"7949852f-3125-40ca-b556-39a355019725\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") "
Jul 28 22:11:46.159164 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159007    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-trusted-ca-bundle\") pod \"7949852f-3125-40ca-b556-39a355019725\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") "
Jul 28 22:11:46.159164 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159040    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-console-config\") pod \"7949852f-3125-40ca-b556-39a355019725\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") "
Jul 28 22:11:46.159164 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-oauth-config\") pod \"7949852f-3125-40ca-b556-39a355019725\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") "
Jul 28 22:11:46.159164 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159108    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-oauth-serving-cert\") pod \"7949852f-3125-40ca-b556-39a355019725\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") "
Jul 28 22:11:46.159380 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159196    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8244\" (UniqueName: \"kubernetes.io/projected/7949852f-3125-40ca-b556-39a355019725-kube-api-access-l8244\") pod \"7949852f-3125-40ca-b556-39a355019725\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") "
Jul 28 22:11:46.159380 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159219    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-serving-cert\") pod \"7949852f-3125-40ca-b556-39a355019725\" (UID: \"7949852f-3125-40ca-b556-39a355019725\") "
Jul 28 22:11:46.159485 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159394    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-service-ca" (OuterVolumeSpecName: "service-ca") pod "7949852f-3125-40ca-b556-39a355019725" (UID: "7949852f-3125-40ca-b556-39a355019725"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:46.159485 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159439    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-console-config" (OuterVolumeSpecName: "console-config") pod "7949852f-3125-40ca-b556-39a355019725" (UID: "7949852f-3125-40ca-b556-39a355019725"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:46.159566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159477    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7949852f-3125-40ca-b556-39a355019725" (UID: "7949852f-3125-40ca-b556-39a355019725"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:46.159670 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159651    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-console-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:46.159670 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7949852f-3125-40ca-b556-39a355019725" (UID: "7949852f-3125-40ca-b556-39a355019725"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:46.159801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.159672    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:46.161327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.161299    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7949852f-3125-40ca-b556-39a355019725-kube-api-access-l8244" (OuterVolumeSpecName: "kube-api-access-l8244") pod "7949852f-3125-40ca-b556-39a355019725" (UID: "7949852f-3125-40ca-b556-39a355019725"). InnerVolumeSpecName "kube-api-access-l8244". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:11:46.161327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.161318    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7949852f-3125-40ca-b556-39a355019725" (UID: "7949852f-3125-40ca-b556-39a355019725"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:11:46.161471 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.161444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7949852f-3125-40ca-b556-39a355019725" (UID: "7949852f-3125-40ca-b556-39a355019725"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:11:46.260755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.260715    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-serving-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:46.260755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.260747    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-trusted-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:46.260755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.260760    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7949852f-3125-40ca-b556-39a355019725-console-oauth-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:46.261023 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.260772    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7949852f-3125-40ca-b556-39a355019725-oauth-serving-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:46.261023 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:46.260784    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8244\" (UniqueName: \"kubernetes.io/projected/7949852f-3125-40ca-b556-39a355019725-kube-api-access-l8244\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:47.065460 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:47.065432    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff897fdc-55mn8"
Jul 28 22:11:47.088268 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:47.088226    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff897fdc-55mn8"]
Jul 28 22:11:47.090648 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:47.090627    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ff897fdc-55mn8"]
Jul 28 22:11:47.121181 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:47.121148    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:47.221950 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:47.221917    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:47.321331 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:47.321226    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:11:48.406822 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:48.406782    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7949852f-3125-40ca-b556-39a355019725" path="/var/lib/kubelet/pods/7949852f-3125-40ca-b556-39a355019725/volumes"
Jul 28 22:11:55.559888 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:55.559854    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:55.563673 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:55.563652    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8577ff485f-gxjgx"
Jul 28 22:11:57.110380 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.110322    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b8f64896c-rcv87" podUID="ed3aa494-6eab-48ae-98d7-3f1a7abee30c" containerName="console" containerID="cri-o://0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b" gracePeriod=15
Jul 28 22:11:57.351566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.351543    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:57.458899 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.458808    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-oauth-serving-cert\") pod \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") "
Jul 28 22:11:57.458899 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.458853    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-oauth-config\") pod \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") "
Jul 28 22:11:57.458899 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.458875    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-service-ca\") pod \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") "
Jul 28 22:11:57.459193 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.458937    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm4xg\" (UniqueName: \"kubernetes.io/projected/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-kube-api-access-dm4xg\") pod \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") "
Jul 28 22:11:57.459193 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.458983    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-serving-cert\") pod \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") "
Jul 28 22:11:57.459193 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.459030    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-config\") pod \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\" (UID: \"ed3aa494-6eab-48ae-98d7-3f1a7abee30c\") "
Jul 28 22:11:57.459348 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.459226    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ed3aa494-6eab-48ae-98d7-3f1a7abee30c" (UID: "ed3aa494-6eab-48ae-98d7-3f1a7abee30c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:57.459348 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.459331    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-oauth-serving-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:57.459444 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.459357    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-service-ca" (OuterVolumeSpecName: "service-ca") pod "ed3aa494-6eab-48ae-98d7-3f1a7abee30c" (UID: "ed3aa494-6eab-48ae-98d7-3f1a7abee30c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:57.459505 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.459487    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-config" (OuterVolumeSpecName: "console-config") pod "ed3aa494-6eab-48ae-98d7-3f1a7abee30c" (UID: "ed3aa494-6eab-48ae-98d7-3f1a7abee30c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:57.461396 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.461373    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-kube-api-access-dm4xg" (OuterVolumeSpecName: "kube-api-access-dm4xg") pod "ed3aa494-6eab-48ae-98d7-3f1a7abee30c" (UID: "ed3aa494-6eab-48ae-98d7-3f1a7abee30c"). InnerVolumeSpecName "kube-api-access-dm4xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:11:57.461478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.461375    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ed3aa494-6eab-48ae-98d7-3f1a7abee30c" (UID: "ed3aa494-6eab-48ae-98d7-3f1a7abee30c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:11:57.461478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.461433    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ed3aa494-6eab-48ae-98d7-3f1a7abee30c" (UID: "ed3aa494-6eab-48ae-98d7-3f1a7abee30c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:11:57.560717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.560682    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4xg\" (UniqueName: \"kubernetes.io/projected/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-kube-api-access-dm4xg\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:57.560717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.560711    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-serving-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:57.560717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.560720    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:57.560940 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.560729    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-console-oauth-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:57.560940 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:57.560738    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3aa494-6eab-48ae-98d7-3f1a7abee30c-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:11:58.110611 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.110565    2560 generic.go:356] "Generic (PLEG): container finished" podID="ed3aa494-6eab-48ae-98d7-3f1a7abee30c" containerID="0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b" exitCode=0
Jul 28 22:11:58.111090 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.110629    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8f64896c-rcv87"
Jul 28 22:11:58.111090 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.110647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8f64896c-rcv87" event={"ID":"ed3aa494-6eab-48ae-98d7-3f1a7abee30c","Type":"ContainerDied","Data":"0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b"}
Jul 28 22:11:58.111090 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.110725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8f64896c-rcv87" event={"ID":"ed3aa494-6eab-48ae-98d7-3f1a7abee30c","Type":"ContainerDied","Data":"527271f330a18f921b7665e6f30916329966a291a2845ecab2121811fe927dce"}
Jul 28 22:11:58.111090 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.110745    2560 scope.go:117] "RemoveContainer" containerID="0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b"
Jul 28 22:11:58.120191 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.120176    2560 scope.go:117] "RemoveContainer" containerID="0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b"
Jul 28 22:11:58.120475 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:11:58.120456    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b\": container with ID starting with 0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b not found: ID does not exist" containerID="0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b"
Jul 28 22:11:58.120531 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.120485    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b"} err="failed to get container status \"0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b\": rpc error: code = NotFound desc = could not find container \"0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b\": container with ID starting with 0529e73473ff496d6c70c240ba8f19068fc00542ab1a7955016263a7e087428b not found: ID does not exist"
Jul 28 22:11:58.132578 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.132553    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8f64896c-rcv87"]
Jul 28 22:11:58.136518 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.136494    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b8f64896c-rcv87"]
Jul 28 22:11:58.407545 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:11:58.407464    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed3aa494-6eab-48ae-98d7-3f1a7abee30c" path="/var/lib/kubelet/pods/ed3aa494-6eab-48ae-98d7-3f1a7abee30c/volumes"
Jul 28 22:12:17.556190 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:17.556139    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:17.575996 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:17.575969    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:18.213540 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:18.213515    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:26.986567 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:26.986535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:27.087443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:27.087413    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:31.263981 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:31.263950    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 22:12:31.264508 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:31.264280    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="alertmanager" containerID="cri-o://32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6" gracePeriod=120
Jul 28 22:12:31.264508 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:31.264342    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy-metric" containerID="cri-o://155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878" gracePeriod=120
Jul 28 22:12:31.264508 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:31.264393    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="prom-label-proxy" containerID="cri-o://5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a" gracePeriod=120
Jul 28 22:12:31.264508 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:31.264412    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy" containerID="cri-o://d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225" gracePeriod=120
Jul 28 22:12:31.264508 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:31.264400    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy-web" containerID="cri-o://f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e" gracePeriod=120
Jul 28 22:12:31.264508 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:31.264459    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="config-reloader" containerID="cri-o://41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393" gracePeriod=120
Jul 28 22:12:32.258451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:32.258420    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b2e445b-db40-4c23-9cc4-51497416156e" containerID="5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a" exitCode=0
Jul 28 22:12:32.258451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:32.258447    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b2e445b-db40-4c23-9cc4-51497416156e" containerID="d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225" exitCode=0
Jul 28 22:12:32.258451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:32.258452    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b2e445b-db40-4c23-9cc4-51497416156e" containerID="41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393" exitCode=0
Jul 28 22:12:32.258451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:32.258458    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b2e445b-db40-4c23-9cc4-51497416156e" containerID="32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6" exitCode=0
Jul 28 22:12:32.258725 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:32.258494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerDied","Data":"5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a"}
Jul 28 22:12:32.258725 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:32.258557    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerDied","Data":"d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225"}
Jul 28 22:12:32.258725 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:32.258584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerDied","Data":"41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393"}
Jul 28 22:12:32.258725 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:32.258608    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerDied","Data":"32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6"}
Jul 28 22:12:32.999376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:32.999344    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.078567 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078534    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-cluster-tls-config\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.078745 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078587    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-config-out\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.078745 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078622    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-web-config\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.078745 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078643    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.078745 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078676    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-tls-assets\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.078745 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078709    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-metrics-client-ca\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.078745 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078730    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-main-tls\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.079052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078766    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.079052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078801    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-trusted-ca-bundle\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.079052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078828    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-config-volume\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.079052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078856    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-main-db\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.079052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078890    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-web\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.079052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.078938    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrhzl\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-kube-api-access-wrhzl\") pod \"6b2e445b-db40-4c23-9cc4-51497416156e\" (UID: \"6b2e445b-db40-4c23-9cc4-51497416156e\") "
Jul 28 22:12:33.079791 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.079176    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:33.079791 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.079347    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-metrics-client-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.079906 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.079849    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:12:33.080007 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.079980    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:33.082740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.082599    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-kube-api-access-wrhzl" (OuterVolumeSpecName: "kube-api-access-wrhzl") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "kube-api-access-wrhzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:12:33.082740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.082658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:33.082945 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.082859    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:33.082945 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.082861    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:12:33.083049 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.082968    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-config-volume" (OuterVolumeSpecName: "config-volume") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:33.083357 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.083328    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:33.083431 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.083407    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-config-out" (OuterVolumeSpecName: "config-out") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:12:33.083516 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.083495    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:33.086679 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.086656    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:33.093206 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.093186    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-web-config" (OuterVolumeSpecName: "web-config") pod "6b2e445b-db40-4c23-9cc4-51497416156e" (UID: "6b2e445b-db40-4c23-9cc4-51497416156e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:33.180646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180547    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-cluster-tls-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.180646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180592    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-config-out\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.180646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180602    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-web-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.180646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180612    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.180646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180621    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-tls-assets\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.180646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180632    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-main-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.180646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180642    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.180646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180652    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.180646 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180661    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.181023 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180670    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b2e445b-db40-4c23-9cc4-51497416156e-alertmanager-main-db\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.181023 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180679    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b2e445b-db40-4c23-9cc4-51497416156e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.181023 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.180687    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrhzl\" (UniqueName: \"kubernetes.io/projected/6b2e445b-db40-4c23-9cc4-51497416156e-kube-api-access-wrhzl\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:33.271026 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.270991    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b2e445b-db40-4c23-9cc4-51497416156e" containerID="155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878" exitCode=0
Jul 28 22:12:33.271026 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.271018    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b2e445b-db40-4c23-9cc4-51497416156e" containerID="f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e" exitCode=0
Jul 28 22:12:33.271249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.271039    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerDied","Data":"155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878"}
Jul 28 22:12:33.271249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.271078    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.271249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.271135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerDied","Data":"f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e"}
Jul 28 22:12:33.271249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.271163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b2e445b-db40-4c23-9cc4-51497416156e","Type":"ContainerDied","Data":"1bebb3fcec97d82096934faec3916637e4435e2c051a3fd656501e8a2ae14748"}
Jul 28 22:12:33.271249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.271192    2560 scope.go:117] "RemoveContainer" containerID="5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a"
Jul 28 22:12:33.279779 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.279759    2560 scope.go:117] "RemoveContainer" containerID="155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878"
Jul 28 22:12:33.286802 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.286777    2560 scope.go:117] "RemoveContainer" containerID="d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225"
Jul 28 22:12:33.293862 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.293841    2560 scope.go:117] "RemoveContainer" containerID="f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e"
Jul 28 22:12:33.301027 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.301008    2560 scope.go:117] "RemoveContainer" containerID="41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393"
Jul 28 22:12:33.307639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.307615    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 22:12:33.309538 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.309520    2560 scope.go:117] "RemoveContainer" containerID="32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6"
Jul 28 22:12:33.317393 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.317366    2560 scope.go:117] "RemoveContainer" containerID="b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f"
Jul 28 22:12:33.325223 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.325115    2560 scope.go:117] "RemoveContainer" containerID="5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a"
Jul 28 22:12:33.325716 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:33.325679    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a\": container with ID starting with 5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a not found: ID does not exist" containerID="5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a"
Jul 28 22:12:33.325816 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.325728    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a"} err="failed to get container status \"5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a\": rpc error: code = NotFound desc = could not find container \"5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a\": container with ID starting with 5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a not found: ID does not exist"
Jul 28 22:12:33.325816 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.325756    2560 scope.go:117] "RemoveContainer" containerID="155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878"
Jul 28 22:12:33.326141 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:33.326075    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878\": container with ID starting with 155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878 not found: ID does not exist" containerID="155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878"
Jul 28 22:12:33.326141 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.326124    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878"} err="failed to get container status \"155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878\": rpc error: code = NotFound desc = could not find container \"155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878\": container with ID starting with 155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878 not found: ID does not exist"
Jul 28 22:12:33.326141 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.326144    2560 scope.go:117] "RemoveContainer" containerID="d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225"
Jul 28 22:12:33.326448 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:33.326422    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225\": container with ID starting with d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225 not found: ID does not exist" containerID="d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225"
Jul 28 22:12:33.326533 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.326459    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225"} err="failed to get container status \"d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225\": rpc error: code = NotFound desc = could not find container \"d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225\": container with ID starting with d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225 not found: ID does not exist"
Jul 28 22:12:33.326533 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.326481    2560 scope.go:117] "RemoveContainer" containerID="f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e"
Jul 28 22:12:33.326855 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:33.326834    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e\": container with ID starting with f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e not found: ID does not exist" containerID="f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e"
Jul 28 22:12:33.326965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.326859    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e"} err="failed to get container status \"f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e\": rpc error: code = NotFound desc = could not find container \"f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e\": container with ID starting with f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e not found: ID does not exist"
Jul 28 22:12:33.326965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.326874    2560 scope.go:117] "RemoveContainer" containerID="41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393"
Jul 28 22:12:33.327198 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:33.327163    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393\": container with ID starting with 41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393 not found: ID does not exist" containerID="41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393"
Jul 28 22:12:33.327198 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.327201    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393"} err="failed to get container status \"41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393\": rpc error: code = NotFound desc = could not find container \"41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393\": container with ID starting with 41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393 not found: ID does not exist"
Jul 28 22:12:33.327338 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.327216    2560 scope.go:117] "RemoveContainer" containerID="32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6"
Jul 28 22:12:33.327487 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:33.327454    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6\": container with ID starting with 32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6 not found: ID does not exist" containerID="32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6"
Jul 28 22:12:33.327535 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.327483    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6"} err="failed to get container status \"32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6\": rpc error: code = NotFound desc = could not find container \"32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6\": container with ID starting with 32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6 not found: ID does not exist"
Jul 28 22:12:33.327535 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.327503    2560 scope.go:117] "RemoveContainer" containerID="b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f"
Jul 28 22:12:33.327745 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:33.327728    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f\": container with ID starting with b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f not found: ID does not exist" containerID="b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f"
Jul 28 22:12:33.327783 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.327749    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f"} err="failed to get container status \"b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f\": rpc error: code = NotFound desc = could not find container \"b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f\": container with ID starting with b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f not found: ID does not exist"
Jul 28 22:12:33.327783 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.327762    2560 scope.go:117] "RemoveContainer" containerID="5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a"
Jul 28 22:12:33.327894 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.327849    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 22:12:33.327983 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.327966    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a"} err="failed to get container status \"5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a\": rpc error: code = NotFound desc = could not find container \"5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a\": container with ID starting with 5a1a738effc9eb300fc2c0c91290f45e2b45206f3a372b3ed5ae245a79d4940a not found: ID does not exist"
Jul 28 22:12:33.328022 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.327984    2560 scope.go:117] "RemoveContainer" containerID="155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878"
Jul 28 22:12:33.328238 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.328220    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878"} err="failed to get container status \"155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878\": rpc error: code = NotFound desc = could not find container \"155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878\": container with ID starting with 155e84dd991949fe9ab7ac2540bd47f6956b748ab0ad6fb845f8a1a4e4c6f878 not found: ID does not exist"
Jul 28 22:12:33.328238 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.328238    2560 scope.go:117] "RemoveContainer" containerID="d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225"
Jul 28 22:12:33.328458 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.328435    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225"} err="failed to get container status \"d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225\": rpc error: code = NotFound desc = could not find container \"d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225\": container with ID starting with d2f7bb4fec63e7f98983c44c760a202724e2ba5b6f2d21c05b417fd6f59ab225 not found: ID does not exist"
Jul 28 22:12:33.328504 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.328459    2560 scope.go:117] "RemoveContainer" containerID="f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e"
Jul 28 22:12:33.328676 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.328655    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e"} err="failed to get container status \"f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e\": rpc error: code = NotFound desc = could not find container \"f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e\": container with ID starting with f993dbd8014cc7e739f6c967270e78a4b70def5cd779347149c1212443a5a16e not found: ID does not exist"
Jul 28 22:12:33.328725 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.328677    2560 scope.go:117] "RemoveContainer" containerID="41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393"
Jul 28 22:12:33.328926 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.328906    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393"} err="failed to get container status \"41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393\": rpc error: code = NotFound desc = could not find container \"41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393\": container with ID starting with 41b19b1b07df65a28a63d58a3a3dfe9f87ffd076391dfbfd7cc74375217a7393 not found: ID does not exist"
Jul 28 22:12:33.328926 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.328925    2560 scope.go:117] "RemoveContainer" containerID="32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6"
Jul 28 22:12:33.329172 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.329155    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6"} err="failed to get container status \"32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6\": rpc error: code = NotFound desc = could not find container \"32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6\": container with ID starting with 32700ebe981c1331cd766f902f85c21286d59dd251e24d94cb79e7087b1bcef6 not found: ID does not exist"
Jul 28 22:12:33.329172 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.329173    2560 scope.go:117] "RemoveContainer" containerID="b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f"
Jul 28 22:12:33.329372 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.329355    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f"} err="failed to get container status \"b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f\": rpc error: code = NotFound desc = could not find container \"b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f\": container with ID starting with b8b2598547fc21897d9e08c12dcaa101a420ab3bc46097b491cebd1e4969a40f not found: ID does not exist"
Jul 28 22:12:33.375611 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375577    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 22:12:33.375914 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375900    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4165e03-01f3-4449-ac4b-b8140148c143" containerName="console"
Jul 28 22:12:33.375960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375917    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4165e03-01f3-4449-ac4b-b8140148c143" containerName="console"
Jul 28 22:12:33.375960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375927    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="init-config-reloader"
Jul 28 22:12:33.375960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375933    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="init-config-reloader"
Jul 28 22:12:33.375960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375939    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy-web"
Jul 28 22:12:33.375960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375945    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy-web"
Jul 28 22:12:33.375960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375953    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7949852f-3125-40ca-b556-39a355019725" containerName="console"
Jul 28 22:12:33.375960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375958    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7949852f-3125-40ca-b556-39a355019725" containerName="console"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375965    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="config-reloader"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375971    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="config-reloader"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375979    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3aa494-6eab-48ae-98d7-3f1a7abee30c" containerName="console"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375984    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3aa494-6eab-48ae-98d7-3f1a7abee30c" containerName="console"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375989    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="alertmanager"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.375994    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="alertmanager"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376002    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376007    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376017    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="prom-label-proxy"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376022    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="prom-label-proxy"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376031    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy-metric"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376035    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy-metric"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376084    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="alertmanager"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376093    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy-metric"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376114    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4165e03-01f3-4449-ac4b-b8140148c143" containerName="console"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376120    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="config-reloader"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376126    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3aa494-6eab-48ae-98d7-3f1a7abee30c" containerName="console"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376131    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="prom-label-proxy"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376139    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376145    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7949852f-3125-40ca-b556-39a355019725" containerName="console"
Jul 28 22:12:33.376246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.376149    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" containerName="kube-rbac-proxy-web"
Jul 28 22:12:33.380901 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.380877    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.383689 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.383665    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 22:12:33.383896 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.383869    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ctvxd\""
Jul 28 22:12:33.384044 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.383898    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 22:12:33.384044 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.383913    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 22:12:33.384044 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.383903    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 22:12:33.384044 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.383945    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 22:12:33.384044 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.384033    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 22:12:33.384358 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.384081    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 22:12:33.384401 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.384360    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 22:12:33.389491 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.389470    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 22:12:33.397708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.397616    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 22:12:33.483698 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.483857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.483857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-config-volume\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.483857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8b3885b-8def-406e-805f-98264697d736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.483857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8b3885b-8def-406e-805f-98264697d736-config-out\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.483857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.484081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8b3885b-8def-406e-805f-98264697d736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.484081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.484081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8b3885b-8def-406e-805f-98264697d736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.484081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.483996    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.484081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.484035    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-web-config\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.484081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.484057    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkc7\" (UniqueName: \"kubernetes.io/projected/d8b3885b-8def-406e-805f-98264697d736-kube-api-access-fdkc7\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.484081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.484078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8b3885b-8def-406e-805f-98264697d736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585001 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.584949    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585001 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-config-volume\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8b3885b-8def-406e-805f-98264697d736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8b3885b-8def-406e-805f-98264697d736-config-out\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585070    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8b3885b-8def-406e-805f-98264697d736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8b3885b-8def-406e-805f-98264697d736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-web-config\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkc7\" (UniqueName: \"kubernetes.io/projected/d8b3885b-8def-406e-805f-98264697d736-kube-api-access-fdkc7\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585766 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8b3885b-8def-406e-805f-98264697d736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585766 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.585766 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8b3885b-8def-406e-805f-98264697d736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.586012 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.585987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8b3885b-8def-406e-805f-98264697d736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.588200 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.587961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8b3885b-8def-406e-805f-98264697d736-config-out\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.588200 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.588147    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-config-volume\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.588374 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.588273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.588374 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.588370    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.588475 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.588372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-web-config\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.588670 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.588648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8b3885b-8def-406e-805f-98264697d736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.588772 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.588748    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.589000 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.588976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8b3885b-8def-406e-805f-98264697d736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.589132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.589091    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.590253 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.590232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8b3885b-8def-406e-805f-98264697d736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.596807 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.596784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkc7\" (UniqueName: \"kubernetes.io/projected/d8b3885b-8def-406e-805f-98264697d736-kube-api-access-fdkc7\") pod \"alertmanager-main-0\" (UID: \"d8b3885b-8def-406e-805f-98264697d736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.690875 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.690836    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 22:12:33.831038 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:33.831009    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 22:12:33.833614 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:12:33.833586    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b3885b_8def_406e_805f_98264697d736.slice/crio-7a8af85bb6445de950f04058dfc9c269d2c36dff0ab2d03222f45536c7a9f428 WatchSource:0}: Error finding container 7a8af85bb6445de950f04058dfc9c269d2c36dff0ab2d03222f45536c7a9f428: Status 404 returned error can't find the container with id 7a8af85bb6445de950f04058dfc9c269d2c36dff0ab2d03222f45536c7a9f428
Jul 28 22:12:34.277241 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:34.277208    2560 generic.go:356] "Generic (PLEG): container finished" podID="d8b3885b-8def-406e-805f-98264697d736" containerID="6c26c34adb79bb4ec1e25126631294cc4dc84ab74d05b358ce4b4913823283f8" exitCode=0
Jul 28 22:12:34.277665 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:34.277267    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8b3885b-8def-406e-805f-98264697d736","Type":"ContainerDied","Data":"6c26c34adb79bb4ec1e25126631294cc4dc84ab74d05b358ce4b4913823283f8"}
Jul 28 22:12:34.277665 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:34.277332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8b3885b-8def-406e-805f-98264697d736","Type":"ContainerStarted","Data":"7a8af85bb6445de950f04058dfc9c269d2c36dff0ab2d03222f45536c7a9f428"}
Jul 28 22:12:34.405865 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:34.405836    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b2e445b-db40-4c23-9cc4-51497416156e" path="/var/lib/kubelet/pods/6b2e445b-db40-4c23-9cc4-51497416156e/volumes"
Jul 28 22:12:35.285670 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.285632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8b3885b-8def-406e-805f-98264697d736","Type":"ContainerStarted","Data":"2be3ffca653ead6229497ed0ee085a6cd85f6d6f1e56146a3a42a65948bd5085"}
Jul 28 22:12:35.286145 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.285747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8b3885b-8def-406e-805f-98264697d736","Type":"ContainerStarted","Data":"3b94db74802c41e080c7b5d873bda5530e458deb616a24244803b2df19ab0574"}
Jul 28 22:12:35.286145 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.285791    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8b3885b-8def-406e-805f-98264697d736","Type":"ContainerStarted","Data":"a03c70d457fb78809829e9c084f39f289b0dd5267a0a8a710bace5edf54987c2"}
Jul 28 22:12:35.286145 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.285869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8b3885b-8def-406e-805f-98264697d736","Type":"ContainerStarted","Data":"ee0b4f59ce3d1856eb52834a7c6312f2d6c33b03b6a5ed99789c6194300f6acf"}
Jul 28 22:12:35.286145 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.285908    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8b3885b-8def-406e-805f-98264697d736","Type":"ContainerStarted","Data":"1ecd70b2b91ffb24e8c8bc54d4961affe4c30087a4489792664ccf84b9c56501"}
Jul 28 22:12:35.286145 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.285950    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8b3885b-8def-406e-805f-98264697d736","Type":"ContainerStarted","Data":"0b3fa911d76d16f6ebac1e6d80dab81f9c7f3bf5941b74eb3b538c92df76bb87"}
Jul 28 22:12:35.316837 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.316467    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.302327217 podStartE2EDuration="2.316445596s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.118379ms" isStatefulPod=false podCreationTimestamp="2026-07-28 22:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:12:35.31497381 +0000 UTC m=+157.437321719" watchObservedRunningTime="2026-07-28 22:12:35.316445596 +0000 UTC m=+157.438793492"
Jul 28 22:12:35.520822 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.520777    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 22:12:35.521298 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.521241    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="prometheus" containerID="cri-o://6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842" gracePeriod=600
Jul 28 22:12:35.521298 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.521264    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy" containerID="cri-o://35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3" gracePeriod=600
Jul 28 22:12:35.521514 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.521315    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="config-reloader" containerID="cri-o://c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a" gracePeriod=600
Jul 28 22:12:35.521514 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.521305    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy-web" containerID="cri-o://abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9" gracePeriod=600
Jul 28 22:12:35.521514 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.521269    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="thanos-sidecar" containerID="cri-o://3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89" gracePeriod=600
Jul 28 22:12:35.521514 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.521297    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838" gracePeriod=600
Jul 28 22:12:35.778187 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.778161    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:35.907940 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.907852    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.907940 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.907890    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-kube-rbac-proxy\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.907940 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.907916    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-trusted-ca-bundle\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.907940 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.907931    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-0\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.907967    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-grpc-tls\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908091    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-metrics-client-ca\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908192    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-tls\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908226    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-config-out\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908255    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-web-config\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908279    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skmhp\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-kube-api-access-skmhp\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908310    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-thanos-prometheus-http-client-file\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908338    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908352    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:35.908639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908362    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-db\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908416    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-serving-certs-ca-bundle\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908445    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-tls-assets\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908487    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-kubelet-serving-ca-bundle\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908603    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-metrics-client-certs\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908982 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908643    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-1\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908982 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908689    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-config\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.908982 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908705    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:35.908982 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.908726    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-2\") pod \"b8b3e424-3036-48bf-9076-a1316c8312a0\" (UID: \"b8b3e424-3036-48bf-9076-a1316c8312a0\") "
Jul 28 22:12:35.909205 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.909003    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:35.909205 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.909139    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:35.909205 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.909184    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:35.909205 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.909202    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-metrics-client-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:35.909403 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.909216    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:35.909403 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.909335    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:35.909558 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.909521    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:12:35.909902 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.909868    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:35.910659 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.910623    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:35.911132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.911075    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:35.911635 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.911608    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:35.911720 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.911680    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:35.911866 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.911839    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:12:35.911991 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.911952    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:35.912490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.912463    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:35.912898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.912874    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-kube-api-access-skmhp" (OuterVolumeSpecName: "kube-api-access-skmhp") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "kube-api-access-skmhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:12:35.913378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.913353    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-config-out" (OuterVolumeSpecName: "config-out") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:12:35.913554 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.913539    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-config" (OuterVolumeSpecName: "config") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:35.913615 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.913574    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:35.913688 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.913669    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:35.922333 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:35.922313    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-web-config" (OuterVolumeSpecName: "web-config") pod "b8b3e424-3036-48bf-9076-a1316c8312a0" (UID: "b8b3e424-3036-48bf-9076-a1316c8312a0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:36.010478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010444    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010474    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-kube-rbac-proxy\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010484    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010495    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-grpc-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010506    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010514    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-config-out\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010523    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-web-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010530    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skmhp\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-kube-api-access-skmhp\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010539    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010549    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010557    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-db\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010565    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010574    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8b3e424-3036-48bf-9076-a1316c8312a0-tls-assets\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010583    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010590    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-secret-metrics-client-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010598    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8b3e424-3036-48bf-9076-a1316c8312a0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.010699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.010606    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8b3e424-3036-48bf-9076-a1316c8312a0-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:12:36.292740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292708    2560 generic.go:356] "Generic (PLEG): container finished" podID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerID="4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838" exitCode=0
Jul 28 22:12:36.292740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292732    2560 generic.go:356] "Generic (PLEG): container finished" podID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerID="35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3" exitCode=0
Jul 28 22:12:36.292740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292740    2560 generic.go:356] "Generic (PLEG): container finished" podID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerID="abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9" exitCode=0
Jul 28 22:12:36.292740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292746    2560 generic.go:356] "Generic (PLEG): container finished" podID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerID="3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89" exitCode=0
Jul 28 22:12:36.292740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292751    2560 generic.go:356] "Generic (PLEG): container finished" podID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerID="c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a" exitCode=0
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292757    2560 generic.go:356] "Generic (PLEG): container finished" podID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerID="6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842" exitCode=0
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerDied","Data":"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"}
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292783    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292825    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerDied","Data":"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"}
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerDied","Data":"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"}
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292880    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerDied","Data":"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"}
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292898    2560 scope.go:117] "RemoveContainer" containerID="4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.292904    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerDied","Data":"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"}
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.293015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerDied","Data":"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"}
Jul 28 22:12:36.293355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.293048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8b3e424-3036-48bf-9076-a1316c8312a0","Type":"ContainerDied","Data":"8327d07a88f562f860340fd2b8a7de0cbcb7fbfa7e2ca4f3236a9d07e8c926dd"}
Jul 28 22:12:36.302383 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.302354    2560 scope.go:117] "RemoveContainer" containerID="35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"
Jul 28 22:12:36.309652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.309638    2560 scope.go:117] "RemoveContainer" containerID="abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"
Jul 28 22:12:36.316352 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.316333    2560 scope.go:117] "RemoveContainer" containerID="3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"
Jul 28 22:12:36.320304 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.320283    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 22:12:36.323507 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.323478    2560 scope.go:117] "RemoveContainer" containerID="c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"
Jul 28 22:12:36.327092 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.327068    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 22:12:36.330621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.330599    2560 scope.go:117] "RemoveContainer" containerID="6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"
Jul 28 22:12:36.337496 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.337479    2560 scope.go:117] "RemoveContainer" containerID="606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"
Jul 28 22:12:36.344083 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.344064    2560 scope.go:117] "RemoveContainer" containerID="4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"
Jul 28 22:12:36.344383 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:36.344362    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": container with ID starting with 4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838 not found: ID does not exist" containerID="4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"
Jul 28 22:12:36.344472 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.344390    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"} err="failed to get container status \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": rpc error: code = NotFound desc = could not find container \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": container with ID starting with 4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838 not found: ID does not exist"
Jul 28 22:12:36.344472 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.344411    2560 scope.go:117] "RemoveContainer" containerID="35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"
Jul 28 22:12:36.344650 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:36.344634    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": container with ID starting with 35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3 not found: ID does not exist" containerID="35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"
Jul 28 22:12:36.344691 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.344654    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"} err="failed to get container status \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": rpc error: code = NotFound desc = could not find container \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": container with ID starting with 35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3 not found: ID does not exist"
Jul 28 22:12:36.344691 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.344666    2560 scope.go:117] "RemoveContainer" containerID="abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"
Jul 28 22:12:36.344873 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:36.344846    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": container with ID starting with abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9 not found: ID does not exist" containerID="abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"
Jul 28 22:12:36.344972 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.344874    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"} err="failed to get container status \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": rpc error: code = NotFound desc = could not find container \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": container with ID starting with abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9 not found: ID does not exist"
Jul 28 22:12:36.344972 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.344899    2560 scope.go:117] "RemoveContainer" containerID="3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"
Jul 28 22:12:36.345120 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:36.345083    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": container with ID starting with 3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89 not found: ID does not exist" containerID="3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"
Jul 28 22:12:36.345188 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.345122    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"} err="failed to get container status \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": rpc error: code = NotFound desc = could not find container \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": container with ID starting with 3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89 not found: ID does not exist"
Jul 28 22:12:36.345188 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.345143    2560 scope.go:117] "RemoveContainer" containerID="c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"
Jul 28 22:12:36.345380 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:36.345363    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": container with ID starting with c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a not found: ID does not exist" containerID="c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"
Jul 28 22:12:36.345450 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.345386    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"} err="failed to get container status \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": rpc error: code = NotFound desc = could not find container \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": container with ID starting with c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a not found: ID does not exist"
Jul 28 22:12:36.345450 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.345404    2560 scope.go:117] "RemoveContainer" containerID="6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"
Jul 28 22:12:36.345642 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:36.345624    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": container with ID starting with 6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842 not found: ID does not exist" containerID="6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"
Jul 28 22:12:36.345680 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.345647    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"} err="failed to get container status \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": rpc error: code = NotFound desc = could not find container \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": container with ID starting with 6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842 not found: ID does not exist"
Jul 28 22:12:36.345680 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.345661    2560 scope.go:117] "RemoveContainer" containerID="606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"
Jul 28 22:12:36.345886 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:12:36.345868    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": container with ID starting with 606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6 not found: ID does not exist" containerID="606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"
Jul 28 22:12:36.345923 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.345890    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"} err="failed to get container status \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": rpc error: code = NotFound desc = could not find container \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": container with ID starting with 606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6 not found: ID does not exist"
Jul 28 22:12:36.345923 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.345903    2560 scope.go:117] "RemoveContainer" containerID="4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"
Jul 28 22:12:36.346119 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.346085    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"} err="failed to get container status \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": rpc error: code = NotFound desc = could not find container \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": container with ID starting with 4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838 not found: ID does not exist"
Jul 28 22:12:36.346195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.346117    2560 scope.go:117] "RemoveContainer" containerID="35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"
Jul 28 22:12:36.346337 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.346319    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"} err="failed to get container status \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": rpc error: code = NotFound desc = could not find container \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": container with ID starting with 35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3 not found: ID does not exist"
Jul 28 22:12:36.346388 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.346338    2560 scope.go:117] "RemoveContainer" containerID="abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"
Jul 28 22:12:36.346552 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.346534    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"} err="failed to get container status \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": rpc error: code = NotFound desc = could not find container \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": container with ID starting with abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9 not found: ID does not exist"
Jul 28 22:12:36.346607 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.346553    2560 scope.go:117] "RemoveContainer" containerID="3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"
Jul 28 22:12:36.346757 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.346738    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"} err="failed to get container status \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": rpc error: code = NotFound desc = could not find container \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": container with ID starting with 3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89 not found: ID does not exist"
Jul 28 22:12:36.346828 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.346758    2560 scope.go:117] "RemoveContainer" containerID="c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"
Jul 28 22:12:36.347004 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.346987    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"} err="failed to get container status \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": rpc error: code = NotFound desc = could not find container \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": container with ID starting with c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a not found: ID does not exist"
Jul 28 22:12:36.347063 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.347004    2560 scope.go:117] "RemoveContainer" containerID="6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"
Jul 28 22:12:36.347223 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.347206    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"} err="failed to get container status \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": rpc error: code = NotFound desc = could not find container \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": container with ID starting with 6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842 not found: ID does not exist"
Jul 28 22:12:36.347267 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.347225    2560 scope.go:117] "RemoveContainer" containerID="606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"
Jul 28 22:12:36.347439 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.347423    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"} err="failed to get container status \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": rpc error: code = NotFound desc = could not find container \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": container with ID starting with 606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6 not found: ID does not exist"
Jul 28 22:12:36.347479 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.347439    2560 scope.go:117] "RemoveContainer" containerID="4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"
Jul 28 22:12:36.347624 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.347607    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"} err="failed to get container status \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": rpc error: code = NotFound desc = could not find container \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": container with ID starting with 4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838 not found: ID does not exist"
Jul 28 22:12:36.347690 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.347627    2560 scope.go:117] "RemoveContainer" containerID="35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"
Jul 28 22:12:36.347867 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.347850    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"} err="failed to get container status \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": rpc error: code = NotFound desc = could not find container \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": container with ID starting with 35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3 not found: ID does not exist"
Jul 28 22:12:36.347916 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.347869    2560 scope.go:117] "RemoveContainer" containerID="abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"
Jul 28 22:12:36.348049 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348031    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"} err="failed to get container status \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": rpc error: code = NotFound desc = could not find container \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": container with ID starting with abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9 not found: ID does not exist"
Jul 28 22:12:36.348131 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348050    2560 scope.go:117] "RemoveContainer" containerID="3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"
Jul 28 22:12:36.348302 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348286    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"} err="failed to get container status \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": rpc error: code = NotFound desc = could not find container \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": container with ID starting with 3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89 not found: ID does not exist"
Jul 28 22:12:36.348346 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348303    2560 scope.go:117] "RemoveContainer" containerID="c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"
Jul 28 22:12:36.348498 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348479    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"} err="failed to get container status \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": rpc error: code = NotFound desc = could not find container \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": container with ID starting with c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a not found: ID does not exist"
Jul 28 22:12:36.348571 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348500    2560 scope.go:117] "RemoveContainer" containerID="6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"
Jul 28 22:12:36.348712 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348695    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"} err="failed to get container status \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": rpc error: code = NotFound desc = could not find container \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": container with ID starting with 6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842 not found: ID does not exist"
Jul 28 22:12:36.348755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348713    2560 scope.go:117] "RemoveContainer" containerID="606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"
Jul 28 22:12:36.348886 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348871    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"} err="failed to get container status \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": rpc error: code = NotFound desc = could not find container \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": container with ID starting with 606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6 not found: ID does not exist"
Jul 28 22:12:36.348928 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.348887    2560 scope.go:117] "RemoveContainer" containerID="4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"
Jul 28 22:12:36.349151 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349093    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"} err="failed to get container status \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": rpc error: code = NotFound desc = could not find container \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": container with ID starting with 4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838 not found: ID does not exist"
Jul 28 22:12:36.349231 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349152    2560 scope.go:117] "RemoveContainer" containerID="35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"
Jul 28 22:12:36.349376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349360    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"} err="failed to get container status \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": rpc error: code = NotFound desc = could not find container \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": container with ID starting with 35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3 not found: ID does not exist"
Jul 28 22:12:36.349413 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349376    2560 scope.go:117] "RemoveContainer" containerID="abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"
Jul 28 22:12:36.349564 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349547    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"} err="failed to get container status \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": rpc error: code = NotFound desc = could not find container \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": container with ID starting with abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9 not found: ID does not exist"
Jul 28 22:12:36.349605 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349566    2560 scope.go:117] "RemoveContainer" containerID="3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"
Jul 28 22:12:36.349786 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349770    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"} err="failed to get container status \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": rpc error: code = NotFound desc = could not find container \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": container with ID starting with 3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89 not found: ID does not exist"
Jul 28 22:12:36.349847 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349787    2560 scope.go:117] "RemoveContainer" containerID="c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"
Jul 28 22:12:36.349984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349963    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"} err="failed to get container status \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": rpc error: code = NotFound desc = could not find container \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": container with ID starting with c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a not found: ID does not exist"
Jul 28 22:12:36.350030 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.349985    2560 scope.go:117] "RemoveContainer" containerID="6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"
Jul 28 22:12:36.350212 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.350195    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"} err="failed to get container status \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": rpc error: code = NotFound desc = could not find container \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": container with ID starting with 6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842 not found: ID does not exist"
Jul 28 22:12:36.350276 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.350214    2560 scope.go:117] "RemoveContainer" containerID="606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"
Jul 28 22:12:36.350437 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.350403    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"} err="failed to get container status \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": rpc error: code = NotFound desc = could not find container \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": container with ID starting with 606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6 not found: ID does not exist"
Jul 28 22:12:36.350482 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.350438    2560 scope.go:117] "RemoveContainer" containerID="4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"
Jul 28 22:12:36.350626 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.350603    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"} err="failed to get container status \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": rpc error: code = NotFound desc = could not find container \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": container with ID starting with 4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838 not found: ID does not exist"
Jul 28 22:12:36.350664 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.350627    2560 scope.go:117] "RemoveContainer" containerID="35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"
Jul 28 22:12:36.350859 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.350843    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"} err="failed to get container status \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": rpc error: code = NotFound desc = could not find container \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": container with ID starting with 35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3 not found: ID does not exist"
Jul 28 22:12:36.350928 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.350861    2560 scope.go:117] "RemoveContainer" containerID="abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"
Jul 28 22:12:36.351058 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.351042    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"} err="failed to get container status \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": rpc error: code = NotFound desc = could not find container \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": container with ID starting with abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9 not found: ID does not exist"
Jul 28 22:12:36.351158 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.351058    2560 scope.go:117] "RemoveContainer" containerID="3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"
Jul 28 22:12:36.351290 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.351264    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"} err="failed to get container status \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": rpc error: code = NotFound desc = could not find container \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": container with ID starting with 3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89 not found: ID does not exist"
Jul 28 22:12:36.351379 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.351290    2560 scope.go:117] "RemoveContainer" containerID="c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"
Jul 28 22:12:36.351675 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.351634    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"} err="failed to get container status \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": rpc error: code = NotFound desc = could not find container \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": container with ID starting with c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a not found: ID does not exist"
Jul 28 22:12:36.351675 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.351660    2560 scope.go:117] "RemoveContainer" containerID="6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"
Jul 28 22:12:36.351941 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.351915    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"} err="failed to get container status \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": rpc error: code = NotFound desc = could not find container \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": container with ID starting with 6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842 not found: ID does not exist"
Jul 28 22:12:36.351941 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.351942    2560 scope.go:117] "RemoveContainer" containerID="606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"
Jul 28 22:12:36.352349 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.352324    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"} err="failed to get container status \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": rpc error: code = NotFound desc = could not find container \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": container with ID starting with 606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6 not found: ID does not exist"
Jul 28 22:12:36.352349 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.352350    2560 scope.go:117] "RemoveContainer" containerID="4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"
Jul 28 22:12:36.352653 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.352632    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838"} err="failed to get container status \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": rpc error: code = NotFound desc = could not find container \"4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838\": container with ID starting with 4848756ed25a1ab1dec15295339a15ac0ee55e3495f8784da5146f7b05b74838 not found: ID does not exist"
Jul 28 22:12:36.352707 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.352655    2560 scope.go:117] "RemoveContainer" containerID="35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"
Jul 28 22:12:36.352900 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.352882    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3"} err="failed to get container status \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": rpc error: code = NotFound desc = could not find container \"35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3\": container with ID starting with 35206e056e775e77a432a5c63d6f2c9df07111922567e4d7f84d6e73b69451c3 not found: ID does not exist"
Jul 28 22:12:36.352973 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.352912    2560 scope.go:117] "RemoveContainer" containerID="abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"
Jul 28 22:12:36.353171 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353150    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9"} err="failed to get container status \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": rpc error: code = NotFound desc = could not find container \"abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9\": container with ID starting with abe46f55b5a974c7324cc6d09dda40e81a1fd506be734b26e63ce1e4d48891c9 not found: ID does not exist"
Jul 28 22:12:36.353224 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353172    2560 scope.go:117] "RemoveContainer" containerID="3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"
Jul 28 22:12:36.353387 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353369    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89"} err="failed to get container status \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": rpc error: code = NotFound desc = could not find container \"3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89\": container with ID starting with 3e26f7109c77decb6177575a967193e62f5df7e164700d9a6a406d2e21178e89 not found: ID does not exist"
Jul 28 22:12:36.353454 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353389    2560 scope.go:117] "RemoveContainer" containerID="c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"
Jul 28 22:12:36.353491 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353451    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 22:12:36.353630 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353611    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a"} err="failed to get container status \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": rpc error: code = NotFound desc = could not find container \"c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a\": container with ID starting with c068b1ceb05421c264ed93dfad9769f44577bbd9f03f757c5a18555683d4d38a not found: ID does not exist"
Jul 28 22:12:36.353693 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353633    2560 scope.go:117] "RemoveContainer" containerID="6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"
Jul 28 22:12:36.353795 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353782    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="config-reloader"
Jul 28 22:12:36.353850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353799    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="config-reloader"
Jul 28 22:12:36.353850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353809    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy"
Jul 28 22:12:36.353850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353816    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy"
Jul 28 22:12:36.353850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353832    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy-web"
Jul 28 22:12:36.353850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353840    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy-web"
Jul 28 22:12:36.353850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353842    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842"} err="failed to get container status \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": rpc error: code = NotFound desc = could not find container \"6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842\": container with ID starting with 6bfea1241150fad7c65911aed17ffac160e8feea744f29f855fbd8477a00a842 not found: ID does not exist"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353860    2560 scope.go:117] "RemoveContainer" containerID="606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353851    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="prometheus"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353907    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="prometheus"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353920    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="init-config-reloader"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353929    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="init-config-reloader"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353943    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy-thanos"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353952    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy-thanos"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353968    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="thanos-sidecar"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.353974    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="thanos-sidecar"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.354057    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="prometheus"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.354067    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy-web"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.354073    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy-thanos"
Jul 28 22:12:36.354082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.354079    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="config-reloader"
Jul 28 22:12:36.354459 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.354092    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="thanos-sidecar"
Jul 28 22:12:36.354459 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.354119    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" containerName="kube-rbac-proxy"
Jul 28 22:12:36.354459 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.354125    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6"} err="failed to get container status \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": rpc error: code = NotFound desc = could not find container \"606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6\": container with ID starting with 606e6faa37b137a00eb8cb9f7f65e8e8c85bbdbf9c9e9c0255c77ba5ce88e4b6 not found: ID does not exist"
Jul 28 22:12:36.358401 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.358362    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.361457 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.361436    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 22:12:36.361688 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.361669    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 22:12:36.361773 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.361699    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6lj5v7fcke3v1\""
Jul 28 22:12:36.361847 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.361826    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 22:12:36.361943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.361912    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 22:12:36.362016 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.361994    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 22:12:36.362078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.362042    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 22:12:36.362150 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.362077    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 22:12:36.362333 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.362311    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 22:12:36.362654 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.362447    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 22:12:36.362654 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.362460    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 22:12:36.362966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.362833    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 22:12:36.363446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.362990    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5s22m\""
Jul 28 22:12:36.363446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.363136    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 22:12:36.364697 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.364676    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 22:12:36.370891 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.368772    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 22:12:36.374072 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.374052    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 22:12:36.405862 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.405819    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b3e424-3036-48bf-9076-a1316c8312a0" path="/var/lib/kubelet/pods/b8b3e424-3036-48bf-9076-a1316c8312a0/volumes"
Jul 28 22:12:36.514951 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.514915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515153 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.514961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515153 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515036    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515153 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-config\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f100bf34-981b-4725-8a12-0cc83cad2040-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kb72\" (UniqueName: \"kubernetes.io/projected/f100bf34-981b-4725-8a12-0cc83cad2040-kube-api-access-5kb72\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515293    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-web-config\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515355    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515392    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515412    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f100bf34-981b-4725-8a12-0cc83cad2040-config-out\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515469    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515817 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.515817 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.515559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616389 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616389 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-web-config\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616389 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616389 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616691 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616691 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616796 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616796 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616905 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f100bf34-981b-4725-8a12-0cc83cad2040-config-out\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616905 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616905 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616857    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.616905 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.616996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.617031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-config\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.617119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f100bf34-981b-4725-8a12-0cc83cad2040-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617365 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.617153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617365 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.617180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617365 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.617211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kb72\" (UniqueName: \"kubernetes.io/projected/f100bf34-981b-4725-8a12-0cc83cad2040-kube-api-access-5kb72\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617365 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.617213    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.617615 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.617589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.619520 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.619493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.619657 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.619632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.619884 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.619861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-web-config\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.619952 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.619882    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-config\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.621195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.620305    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.621195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.620470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f100bf34-981b-4725-8a12-0cc83cad2040-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.621195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.620468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.621195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.620568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.621195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.620923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.621195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.620991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.621195 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.621174    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.621685 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.621568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.621743 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.621707    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f100bf34-981b-4725-8a12-0cc83cad2040-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.622330 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.622307    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.622429 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.622410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.622603 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.622589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f100bf34-981b-4725-8a12-0cc83cad2040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.622935 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.622921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f100bf34-981b-4725-8a12-0cc83cad2040-config-out\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.625216 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.625196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kb72\" (UniqueName: \"kubernetes.io/projected/f100bf34-981b-4725-8a12-0cc83cad2040-kube-api-access-5kb72\") pod \"prometheus-k8s-0\" (UID: \"f100bf34-981b-4725-8a12-0cc83cad2040\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.682857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.682824    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:36.817728 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:36.817683    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 22:12:36.819788 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:12:36.819762    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf100bf34_981b_4725_8a12_0cc83cad2040.slice/crio-0e7d87c12263845620226d477fb04d9d01eec2892d821dd6c68734cad18f39d0 WatchSource:0}: Error finding container 0e7d87c12263845620226d477fb04d9d01eec2892d821dd6c68734cad18f39d0: Status 404 returned error can't find the container with id 0e7d87c12263845620226d477fb04d9d01eec2892d821dd6c68734cad18f39d0
Jul 28 22:12:37.299868 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:37.299833    2560 generic.go:356] "Generic (PLEG): container finished" podID="f100bf34-981b-4725-8a12-0cc83cad2040" containerID="635ed0b3be2480da27f870b994c2f39fb513e1b8f03985e8b5dd1fdca1bb692b" exitCode=0
Jul 28 22:12:37.300289 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:37.299876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f100bf34-981b-4725-8a12-0cc83cad2040","Type":"ContainerDied","Data":"635ed0b3be2480da27f870b994c2f39fb513e1b8f03985e8b5dd1fdca1bb692b"}
Jul 28 22:12:37.300289 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:37.299935    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f100bf34-981b-4725-8a12-0cc83cad2040","Type":"ContainerStarted","Data":"0e7d87c12263845620226d477fb04d9d01eec2892d821dd6c68734cad18f39d0"}
Jul 28 22:12:38.307774 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:38.307743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f100bf34-981b-4725-8a12-0cc83cad2040","Type":"ContainerStarted","Data":"9ef563be08e43ff645c4aaed60fcbf468d17a77ddec41f108060641fe5b87729"}
Jul 28 22:12:38.308135 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:38.307806    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f100bf34-981b-4725-8a12-0cc83cad2040","Type":"ContainerStarted","Data":"ddb8ee61c8bade7e8f9ab8abfb5aa253232a95d6ad3f267065fcb8fce2e76da3"}
Jul 28 22:12:38.308135 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:38.307837    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f100bf34-981b-4725-8a12-0cc83cad2040","Type":"ContainerStarted","Data":"f423c051cf0e79c892eaaf4e3f54fe4d3ac612bb65032c8988e90ca8bbebdcf0"}
Jul 28 22:12:38.308135 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:38.307862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f100bf34-981b-4725-8a12-0cc83cad2040","Type":"ContainerStarted","Data":"b037a521d4e139df7c519ec87f62ad7453c9b66961c49c6bf0a0c2989e8ca394"}
Jul 28 22:12:38.308135 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:38.307887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f100bf34-981b-4725-8a12-0cc83cad2040","Type":"ContainerStarted","Data":"18c36794acebfc0ed042da9f742fadf02ce5e864c9babea0c8d246231ff2cf87"}
Jul 28 22:12:38.308135 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:38.307910    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f100bf34-981b-4725-8a12-0cc83cad2040","Type":"ContainerStarted","Data":"34fe163221e523cadb8541e93c34b1240bea0447f91a7a6838459d3703bb94a1"}
Jul 28 22:12:38.338227 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:38.338167    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.320523556 podStartE2EDuration="2.3381461s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.622544ms" isStatefulPod=false podCreationTimestamp="2026-07-28 22:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:12:38.335781873 +0000 UTC m=+160.458129787" watchObservedRunningTime="2026-07-28 22:12:38.3381461 +0000 UTC m=+160.460493995"
Jul 28 22:12:38.940824 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:38.940792    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:38.996308 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:38.996275    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:41.683807 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:41.683770    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:12:44.980366 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:44.980333    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:45.081143 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:45.081087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:45.182079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:45.182019    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:45.726966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:45.726933    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:45.778604 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:45.778543    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:51.776543 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:51.776455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:51.879360 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:51.879320    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:54.815174 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:54.815134    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:54.914542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:54.914510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:54.965464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:54.965432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:55.515719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:55.515691    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:55.565281 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:55.565243    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:55.618947 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:55.618922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:12:55.674513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:12:55.674483    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:17.673113 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:17.673058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:17.774128 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:17.774087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:26.141764 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:26.141726    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:26.239871 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:26.239830    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:28.613088 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:28.613048    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:36.683311 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:36.683274    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:13:36.698411 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:36.698382    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:13:37.564283 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:37.564256    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 22:13:47.808255 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:47.808223    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:48.787733 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:48.787696    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:51.407549 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:51.407515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:51.545054 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:51.545020    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:53.676695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:53.676660    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:53.778254 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:53.778213    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:54.009240 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:54.009205    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:54.134312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:54.134277    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:55.876534 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:55.876500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:56.484357 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:56.484328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:58.407530 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:58.407499    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:58.465053 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:58.465018    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:13:58.871992 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:58.871953    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"]
Jul 28 22:13:58.875524 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:58.875503    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"
Jul 28 22:13:58.878152 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:58.878128    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 22:13:58.879186 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:58.879168    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-862t8\""
Jul 28 22:13:58.879303 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:58.879173    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 22:13:58.884454 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:58.884428    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"]
Jul 28 22:13:59.061161 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:59.061124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtr2r\" (UniqueName: \"kubernetes.io/projected/ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40-kube-api-access-vtr2r\") pod \"cert-manager-cainjector-559cdd8c87-2cdhh\" (UID: \"ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"
Jul 28 22:13:59.061321 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:59.061251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2cdhh\" (UID: \"ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"
Jul 28 22:13:59.162078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:59.161978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2cdhh\" (UID: \"ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"
Jul 28 22:13:59.162078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:59.162049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtr2r\" (UniqueName: \"kubernetes.io/projected/ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40-kube-api-access-vtr2r\") pod \"cert-manager-cainjector-559cdd8c87-2cdhh\" (UID: \"ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"
Jul 28 22:13:59.170501 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:59.170478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2cdhh\" (UID: \"ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"
Jul 28 22:13:59.170641 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:59.170571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtr2r\" (UniqueName: \"kubernetes.io/projected/ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40-kube-api-access-vtr2r\") pod \"cert-manager-cainjector-559cdd8c87-2cdhh\" (UID: \"ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"
Jul 28 22:13:59.193547 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:59.193508    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"
Jul 28 22:13:59.321708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:59.321684    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh"]
Jul 28 22:13:59.324519 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:13:59.324481    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab6df819_7ed5_4a06_8a2b_3cc8a2d95c40.slice/crio-2105bedb7a7b4c296b2e03fb9175d0e28d0a71cb028b985491f21e0afb5e45b7 WatchSource:0}: Error finding container 2105bedb7a7b4c296b2e03fb9175d0e28d0a71cb028b985491f21e0afb5e45b7: Status 404 returned error can't find the container with id 2105bedb7a7b4c296b2e03fb9175d0e28d0a71cb028b985491f21e0afb5e45b7
Jul 28 22:13:59.632470 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:13:59.632442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh" event={"ID":"ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40","Type":"ContainerStarted","Data":"2105bedb7a7b4c296b2e03fb9175d0e28d0a71cb028b985491f21e0afb5e45b7"}
Jul 28 22:14:03.650325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:03.650285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh" event={"ID":"ab6df819-7ed5-4a06-8a2b-3cc8a2d95c40","Type":"ContainerStarted","Data":"adb02c6baeef0209e273151d49daaa809612c1743b10d55f1add9c32bea414d4"}
Jul 28 22:14:03.666084 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:03.666033    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2cdhh" podStartSLOduration=2.009723648 podStartE2EDuration="5.666014201s" totalImagesPullingTime="3.656290553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:03.665186165 +0000 UTC m=+245.787534059" watchObservedRunningTime="2026-07-28 22:14:03.666014201 +0000 UTC m=+245.788362094"
Jul 28 22:14:04.478951 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:04.478915    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:06.316965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:06.316932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:06.418150 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:06.418117    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:06.525536 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:06.525474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:06.589756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:06.589670    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:08.351151 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.351093    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"]
Jul 28 22:14:08.354692 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.354671    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"
Jul 28 22:14:08.357197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.357168    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 22:14:08.357303 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.357240    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 22:14:08.358229 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.358217    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-v5c7l\""
Jul 28 22:14:08.364955 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.364933    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"]
Jul 28 22:14:08.438365 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.438335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnq9\" (UniqueName: \"kubernetes.io/projected/6a69e702-574e-4375-96c4-e88faa1d12cb-kube-api-access-2vnq9\") pod \"openshift-lws-operator-fd8ccff4c-974zq\" (UID: \"6a69e702-574e-4375-96c4-e88faa1d12cb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"
Jul 28 22:14:08.438365 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.438368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a69e702-574e-4375-96c4-e88faa1d12cb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-974zq\" (UID: \"6a69e702-574e-4375-96c4-e88faa1d12cb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"
Jul 28 22:14:08.539213 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.539180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnq9\" (UniqueName: \"kubernetes.io/projected/6a69e702-574e-4375-96c4-e88faa1d12cb-kube-api-access-2vnq9\") pod \"openshift-lws-operator-fd8ccff4c-974zq\" (UID: \"6a69e702-574e-4375-96c4-e88faa1d12cb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"
Jul 28 22:14:08.539213 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.539221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a69e702-574e-4375-96c4-e88faa1d12cb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-974zq\" (UID: \"6a69e702-574e-4375-96c4-e88faa1d12cb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"
Jul 28 22:14:08.539595 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.539576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a69e702-574e-4375-96c4-e88faa1d12cb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-974zq\" (UID: \"6a69e702-574e-4375-96c4-e88faa1d12cb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"
Jul 28 22:14:08.548398 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.548373    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnq9\" (UniqueName: \"kubernetes.io/projected/6a69e702-574e-4375-96c4-e88faa1d12cb-kube-api-access-2vnq9\") pod \"openshift-lws-operator-fd8ccff4c-974zq\" (UID: \"6a69e702-574e-4375-96c4-e88faa1d12cb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"
Jul 28 22:14:08.663281 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.663195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"
Jul 28 22:14:08.788410 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:08.788381    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq"]
Jul 28 22:14:08.790951 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:14:08.790922    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a69e702_574e_4375_96c4_e88faa1d12cb.slice/crio-dd5b2dd200397a5253f0f254ca5d18d9bafec918ed6b4c175d6eb9118ad4b78a WatchSource:0}: Error finding container dd5b2dd200397a5253f0f254ca5d18d9bafec918ed6b4c175d6eb9118ad4b78a: Status 404 returned error can't find the container with id dd5b2dd200397a5253f0f254ca5d18d9bafec918ed6b4c175d6eb9118ad4b78a
Jul 28 22:14:09.295744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:09.295710    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:09.350332 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:09.350303    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:09.677492 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:09.677405    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq" event={"ID":"6a69e702-574e-4375-96c4-e88faa1d12cb","Type":"ContainerStarted","Data":"dd5b2dd200397a5253f0f254ca5d18d9bafec918ed6b4c175d6eb9118ad4b78a"}
Jul 28 22:14:11.688784 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:11.688673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq" event={"ID":"6a69e702-574e-4375-96c4-e88faa1d12cb","Type":"ContainerStarted","Data":"5f6c8c76efa8300aa21eab8dae0aa1da34181d9f283f5e697897caf23e84acce"}
Jul 28 22:14:11.703926 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:11.703793    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-974zq" podStartSLOduration=1.197522419 podStartE2EDuration="3.703775142s" totalImagesPullingTime="2.506252723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:11.703196158 +0000 UTC m=+253.825544051" watchObservedRunningTime="2026-07-28 22:14:11.703775142 +0000 UTC m=+253.826123038"
Jul 28 22:14:12.435147 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:12.435089    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:12.536632 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:12.536587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:14.154943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:14.154909    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:27.055320 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:27.055233    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:27.150615 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:27.150582    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:27.665018 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:27.664984    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:27.885423 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:27.885389    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:27.989244 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:27.989213    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:28.061371 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.061336    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"]
Jul 28 22:14:28.067997 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.067972    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.076772 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.076755    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 22:14:28.076877 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.076861    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 22:14:28.077509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.077494    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 22:14:28.077562 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.077503    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zwhdk\""
Jul 28 22:14:28.077741 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.077728    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 22:14:28.096073 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.096049    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"]
Jul 28 22:14:28.213340 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.213298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1d80ea2-cd03-4d00-89b4-df2716812207-manifests\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.213540 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.213349    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgnl4\" (UniqueName: \"kubernetes.io/projected/e1d80ea2-cd03-4d00-89b4-df2716812207-kube-api-access-zgnl4\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.213540 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.213428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1d80ea2-cd03-4d00-89b4-df2716812207-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.213540 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.213455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1d80ea2-cd03-4d00-89b4-df2716812207-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.213540 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.213505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1d80ea2-cd03-4d00-89b4-df2716812207-tmp\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.253825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.253723    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:28.314541 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.314509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1d80ea2-cd03-4d00-89b4-df2716812207-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.314541 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.314541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1d80ea2-cd03-4d00-89b4-df2716812207-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.314763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.314576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1d80ea2-cd03-4d00-89b4-df2716812207-tmp\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.314763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.314631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1d80ea2-cd03-4d00-89b4-df2716812207-manifests\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.314763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.314648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgnl4\" (UniqueName: \"kubernetes.io/projected/e1d80ea2-cd03-4d00-89b4-df2716812207-kube-api-access-zgnl4\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.315132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.315076    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1d80ea2-cd03-4d00-89b4-df2716812207-manifests\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.316915 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.316888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1d80ea2-cd03-4d00-89b4-df2716812207-tmp\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.317022 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.316991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1d80ea2-cd03-4d00-89b4-df2716812207-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.317076 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.316995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1d80ea2-cd03-4d00-89b4-df2716812207-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.325289 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.325270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgnl4\" (UniqueName: \"kubernetes.io/projected/e1d80ea2-cd03-4d00-89b4-df2716812207-kube-api-access-zgnl4\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-7dfws\" (UID: \"e1d80ea2-cd03-4d00-89b4-df2716812207\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.354710 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.354674    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:28.378471 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.378447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:28.459338 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.459248    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:28.539753 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.539663    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"]
Jul 28 22:14:28.544870 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:14:28.544844    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d80ea2_cd03_4d00_89b4_df2716812207.slice/crio-9fdd3b01e519efe888931721c678eb9fa5652a0e4a2d90c64a85ffbcedbee457 WatchSource:0}: Error finding container 9fdd3b01e519efe888931721c678eb9fa5652a0e4a2d90c64a85ffbcedbee457: Status 404 returned error can't find the container with id 9fdd3b01e519efe888931721c678eb9fa5652a0e4a2d90c64a85ffbcedbee457
Jul 28 22:14:28.560824 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.560799    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:28.584512 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.584482    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"]
Jul 28 22:14:28.589786 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.589765    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.594318 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.594294    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 22:14:28.594432 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.594416    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-llnh7\""
Jul 28 22:14:28.594560 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.594545    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 22:14:28.597395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.597379    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 22:14:28.609841 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.609816    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"]
Jul 28 22:14:28.688931 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.688904    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:28.718258 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.718228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47f07dff-d56c-4cf8-a247-0f1c34a28170-cert\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.718378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.718275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9sq6\" (UniqueName: \"kubernetes.io/projected/47f07dff-d56c-4cf8-a247-0f1c34a28170-kube-api-access-v9sq6\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.718378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.718296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47f07dff-d56c-4cf8-a247-0f1c34a28170-metrics-cert\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.718446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.718369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/47f07dff-d56c-4cf8-a247-0f1c34a28170-manager-config\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.757126 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.757073    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws" event={"ID":"e1d80ea2-cd03-4d00-89b4-df2716812207","Type":"ContainerStarted","Data":"9fdd3b01e519efe888931721c678eb9fa5652a0e4a2d90c64a85ffbcedbee457"}
Jul 28 22:14:28.819220 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.819127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9sq6\" (UniqueName: \"kubernetes.io/projected/47f07dff-d56c-4cf8-a247-0f1c34a28170-kube-api-access-v9sq6\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.819220 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.819165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47f07dff-d56c-4cf8-a247-0f1c34a28170-metrics-cert\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.819448 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.819350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/47f07dff-d56c-4cf8-a247-0f1c34a28170-manager-config\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.819448 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.819438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47f07dff-d56c-4cf8-a247-0f1c34a28170-cert\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.819965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.819941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/47f07dff-d56c-4cf8-a247-0f1c34a28170-manager-config\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.821542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.821522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47f07dff-d56c-4cf8-a247-0f1c34a28170-metrics-cert\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.821631 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.821608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47f07dff-d56c-4cf8-a247-0f1c34a28170-cert\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.828618 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.828594    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9sq6\" (UniqueName: \"kubernetes.io/projected/47f07dff-d56c-4cf8-a247-0f1c34a28170-kube-api-access-v9sq6\") pod \"lws-controller-manager-8586db4f6f-q5dd2\" (UID: \"47f07dff-d56c-4cf8-a247-0f1c34a28170\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:28.899964 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:28.899930    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:29.036514 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:29.036488    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"]
Jul 28 22:14:29.038591 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:14:29.038563    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f07dff_d56c_4cf8_a247_0f1c34a28170.slice/crio-8f79ae73748bc73beaa970efa4eac2ccd8f0c09aab6a4a5e7e44be0b2ddd5a30 WatchSource:0}: Error finding container 8f79ae73748bc73beaa970efa4eac2ccd8f0c09aab6a4a5e7e44be0b2ddd5a30: Status 404 returned error can't find the container with id 8f79ae73748bc73beaa970efa4eac2ccd8f0c09aab6a4a5e7e44be0b2ddd5a30
Jul 28 22:14:29.201519 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:29.201413    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:29.302807 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:29.302778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:29.766286 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:29.766216    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2" event={"ID":"47f07dff-d56c-4cf8-a247-0f1c34a28170","Type":"ContainerStarted","Data":"8f79ae73748bc73beaa970efa4eac2ccd8f0c09aab6a4a5e7e44be0b2ddd5a30"}
Jul 28 22:14:32.784385 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:32.784353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2" event={"ID":"47f07dff-d56c-4cf8-a247-0f1c34a28170","Type":"ContainerStarted","Data":"e8eec49e91d30e24ac9b01780004586f5b94945c3765903c81704e28effe23bb"}
Jul 28 22:14:32.785023 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:32.784484    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:32.800568 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:32.800524    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2" podStartSLOduration=1.172943553 podStartE2EDuration="4.80050967s" totalImagesPullingTime="3.627566117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:32.799892543 +0000 UTC m=+274.922240459" watchObservedRunningTime="2026-07-28 22:14:32.80050967 +0000 UTC m=+274.922857595"
Jul 28 22:14:33.790249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:33.790212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws" event={"ID":"e1d80ea2-cd03-4d00-89b4-df2716812207","Type":"ContainerStarted","Data":"ac9dfb18f5f44c36f3aef6ed24c4712ea8b581c0bfd1a55a67ac1470a2dea3ce"}
Jul 28 22:14:34.796181 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:34.796144    2560 generic.go:356] "Generic (PLEG): container finished" podID="e1d80ea2-cd03-4d00-89b4-df2716812207" containerID="ac9dfb18f5f44c36f3aef6ed24c4712ea8b581c0bfd1a55a67ac1470a2dea3ce" exitCode=0
Jul 28 22:14:34.796574 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:34.796235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws" event={"ID":"e1d80ea2-cd03-4d00-89b4-df2716812207","Type":"ContainerDied","Data":"ac9dfb18f5f44c36f3aef6ed24c4712ea8b581c0bfd1a55a67ac1470a2dea3ce"}
Jul 28 22:14:35.802040 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:35.802005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws" event={"ID":"e1d80ea2-cd03-4d00-89b4-df2716812207","Type":"ContainerStarted","Data":"a9f0bb5d9bbea0b0a6473c42a3d16e8a0e4571e91c1fc967b9565e198f433e41"}
Jul 28 22:14:35.802436 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:35.802139    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:35.826075 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:35.826030    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws" podStartSLOduration=1.957782446 podStartE2EDuration="7.826015771s" totalImagesPullingTime="4.823958928s" totalInitContainerRuntime="1.044274397s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:35.824877132 +0000 UTC m=+277.947225042" watchObservedRunningTime="2026-07-28 22:14:35.826015771 +0000 UTC m=+277.948363664"
Jul 28 22:14:42.267912 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:42.267874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:43.792925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:43.792889    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-q5dd2"
Jul 28 22:14:45.021535 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:45.021501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:46.809478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:46.809452    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-7dfws"
Jul 28 22:14:48.391014 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:48.390951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:48.511419 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:48.511388    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:49.058657 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.058614    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:49.119775 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.119748    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"]
Jul 28 22:14:49.125292 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.125271    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.127966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.127945    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 22:14:49.128088 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.128037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-q6nvs\""
Jul 28 22:14:49.128208 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.128186    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 22:14:49.129174 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.129156    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 22:14:49.129274 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.129158    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 22:14:49.135142 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.134950    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"]
Jul 28 22:14:49.202256 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.202221    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwxc\" (UniqueName: \"kubernetes.io/projected/f4727687-705d-4799-a3dc-9a90cb2adf8d-kube-api-access-9jwxc\") pod \"kube-auth-proxy-5d4df5cb96-dl8mh\" (UID: \"f4727687-705d-4799-a3dc-9a90cb2adf8d\") " pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.202431 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.202375    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4727687-705d-4799-a3dc-9a90cb2adf8d-tls-certs\") pod \"kube-auth-proxy-5d4df5cb96-dl8mh\" (UID: \"f4727687-705d-4799-a3dc-9a90cb2adf8d\") " pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.202431 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.202397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4727687-705d-4799-a3dc-9a90cb2adf8d-tmp\") pod \"kube-auth-proxy-5d4df5cb96-dl8mh\" (UID: \"f4727687-705d-4799-a3dc-9a90cb2adf8d\") " pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.302814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.302777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwxc\" (UniqueName: \"kubernetes.io/projected/f4727687-705d-4799-a3dc-9a90cb2adf8d-kube-api-access-9jwxc\") pod \"kube-auth-proxy-5d4df5cb96-dl8mh\" (UID: \"f4727687-705d-4799-a3dc-9a90cb2adf8d\") " pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.302977 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.302906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4727687-705d-4799-a3dc-9a90cb2adf8d-tls-certs\") pod \"kube-auth-proxy-5d4df5cb96-dl8mh\" (UID: \"f4727687-705d-4799-a3dc-9a90cb2adf8d\") " pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.302977 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.302931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4727687-705d-4799-a3dc-9a90cb2adf8d-tmp\") pod \"kube-auth-proxy-5d4df5cb96-dl8mh\" (UID: \"f4727687-705d-4799-a3dc-9a90cb2adf8d\") " pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.305244 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.305215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4727687-705d-4799-a3dc-9a90cb2adf8d-tmp\") pod \"kube-auth-proxy-5d4df5cb96-dl8mh\" (UID: \"f4727687-705d-4799-a3dc-9a90cb2adf8d\") " pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.305417 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.305281    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4727687-705d-4799-a3dc-9a90cb2adf8d-tls-certs\") pod \"kube-auth-proxy-5d4df5cb96-dl8mh\" (UID: \"f4727687-705d-4799-a3dc-9a90cb2adf8d\") " pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.319842 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.319783    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwxc\" (UniqueName: \"kubernetes.io/projected/f4727687-705d-4799-a3dc-9a90cb2adf8d-kube-api-access-9jwxc\") pod \"kube-auth-proxy-5d4df5cb96-dl8mh\" (UID: \"f4727687-705d-4799-a3dc-9a90cb2adf8d\") " pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.436377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.436345    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"
Jul 28 22:14:49.574921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.574834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh"]
Jul 28 22:14:49.578691 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:14:49.578664    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4727687_705d_4799_a3dc_9a90cb2adf8d.slice/crio-e28b90ec6efcad026661a434255b82a65367bd75a3ff5e64ef14a3a74f1a1fe0 WatchSource:0}: Error finding container e28b90ec6efcad026661a434255b82a65367bd75a3ff5e64ef14a3a74f1a1fe0: Status 404 returned error can't find the container with id e28b90ec6efcad026661a434255b82a65367bd75a3ff5e64ef14a3a74f1a1fe0
Jul 28 22:14:49.754313 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.754284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:49.862559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.862458    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:49.874156 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:49.874128    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh" event={"ID":"f4727687-705d-4799-a3dc-9a90cb2adf8d","Type":"ContainerStarted","Data":"e28b90ec6efcad026661a434255b82a65367bd75a3ff5e64ef14a3a74f1a1fe0"}
Jul 28 22:14:50.277780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:50.277739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:50.497961 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:50.497926    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:50.565108 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:50.565001    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:51.191133 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:51.191081    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:51.435798 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:51.435763    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:52.891554 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:52.891517    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh" event={"ID":"f4727687-705d-4799-a3dc-9a90cb2adf8d","Type":"ContainerStarted","Data":"f07b0e4a24a47d8e5f63c91350f5a7e60028896e08f527b01b4129b23affa528"}
Jul 28 22:14:52.910180 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:52.910090    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d4df5cb96-dl8mh" podStartSLOduration=1.485169158 podStartE2EDuration="3.910074316s" totalImagesPullingTime="2.424905158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:52.909185716 +0000 UTC m=+295.031533633" watchObservedRunningTime="2026-07-28 22:14:52.910074316 +0000 UTC m=+295.032422213"
Jul 28 22:14:56.103011 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:56.102980    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:56.205872 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:56.205843    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:56.397990 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:56.397903    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:56.797571 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:56.797534    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:56.896847 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:56.896814    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:14:58.264403 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:14:58.264383    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 22:15:00.859202 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:00.859168    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:00.965838 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:00.965806    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:01.223250 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:01.223213    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:01.335901 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:01.335866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:01.440363 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:01.440327    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:01.513436 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:01.513333    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:07.853119 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:07.849836    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:07.909418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:07.909389    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:49.798991 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:49.798953    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:50.580387 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:50.580352    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:15:50.681574 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:15:50.681536    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:03.069921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:03.069884    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:13.998279 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:13.998223    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:17.213635 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:17.213604    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:17.322127 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:17.322070    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:22.589445 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:22.589395    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:22.825170 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:22.825130    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:25.168159 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:25.168125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:25.417535 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:25.417503    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:25.589432 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:25.589397    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:26.604840 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:26.604802    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:27.206991 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.206957    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66c9496995-kx5qc"]
Jul 28 22:16:27.210673 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.210653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.213159 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.213135    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 22:16:27.214480 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.214461    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 22:16:27.214619 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.214479    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 22:16:27.214708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.214613    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 22:16:27.214708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.214526    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 22:16:27.214708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.214682    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 22:16:27.214878 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.214619    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 22:16:27.214878 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.214496    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jr9lx\""
Jul 28 22:16:27.219376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.219355    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 22:16:27.223870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.223851    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c9496995-kx5qc"]
Jul 28 22:16:27.267343 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.267307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-console-config\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.267537 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.267387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-trusted-ca-bundle\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.267537 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.267465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-console-serving-cert\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.267537 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.267501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-service-ca\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.267719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.267552    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9clbq\" (UniqueName: \"kubernetes.io/projected/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-kube-api-access-9clbq\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.267719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.267578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-oauth-serving-cert\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.267719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.267675    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-console-oauth-config\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.268725 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.268691    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:27.368089 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.368053    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:27.368293 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.368163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-console-config\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.368293 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.368199    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-trusted-ca-bundle\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.368293 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.368239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-console-serving-cert\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.368293 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.368264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-service-ca\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.368293 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.368290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9clbq\" (UniqueName: \"kubernetes.io/projected/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-kube-api-access-9clbq\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.368563 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.368326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-oauth-serving-cert\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.368563 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.368415    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-console-oauth-config\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.369192 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.369165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-console-config\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.369192 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.369176    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-service-ca\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.369192 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.369182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-oauth-serving-cert\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.369446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.369419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-trusted-ca-bundle\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.370864 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.370836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-console-oauth-config\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.371031 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.371014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-console-serving-cert\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.378475 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.378450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9clbq\" (UniqueName: \"kubernetes.io/projected/ed8d3d3c-963e-455b-a6b5-7a2f59c998a4-kube-api-access-9clbq\") pod \"console-66c9496995-kx5qc\" (UID: \"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4\") " pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.521219 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.521178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:27.658261 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.658226    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c9496995-kx5qc"]
Jul 28 22:16:27.661626 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:16:27.661597    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8d3d3c_963e_455b_a6b5_7a2f59c998a4.slice/crio-d494044419237e30bc4346ab96e1b9cf6555c146d045d0ca07f6454917ee5f7c WatchSource:0}: Error finding container d494044419237e30bc4346ab96e1b9cf6555c146d045d0ca07f6454917ee5f7c: Status 404 returned error can't find the container with id d494044419237e30bc4346ab96e1b9cf6555c146d045d0ca07f6454917ee5f7c
Jul 28 22:16:27.807697 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:27.807597    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:28.326265 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.326226    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c9496995-kx5qc" event={"ID":"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4","Type":"ContainerStarted","Data":"d7879208f8a76d2dc43fb5ba3edbab9e37d5599c5da8c8866715e7d757cbffb9"}
Jul 28 22:16:28.326462 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.326279    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c9496995-kx5qc" event={"ID":"ed8d3d3c-963e-455b-a6b5-7a2f59c998a4","Type":"ContainerStarted","Data":"d494044419237e30bc4346ab96e1b9cf6555c146d045d0ca07f6454917ee5f7c"}
Jul 28 22:16:28.347895 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.347842    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66c9496995-kx5qc" podStartSLOduration=1.3478249930000001 podStartE2EDuration="1.347824993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:16:28.346931036 +0000 UTC m=+390.469278942" watchObservedRunningTime="2026-07-28 22:16:28.347824993 +0000 UTC m=+390.470172886"
Jul 28 22:16:28.855474 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.855443    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"]
Jul 28 22:16:28.858928 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.858906    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:28.861608 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.861584    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 22:16:28.861744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.861646    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 22:16:28.861744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.861725    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 22:16:28.861857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.861731    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j2rx6\""
Jul 28 22:16:28.861857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.861782    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 22:16:28.867196 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.867167    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"]
Jul 28 22:16:28.870054 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.870027    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:28.883071 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.883046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7885d1c-f834-4a26-9e45-85e170caa048-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:28.883212 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.883116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c7885d1c-f834-4a26-9e45-85e170caa048-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:28.883254 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.883217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9j4x\" (UniqueName: \"kubernetes.io/projected/c7885d1c-f834-4a26-9e45-85e170caa048-kube-api-access-k9j4x\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:28.984384 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.984349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9j4x\" (UniqueName: \"kubernetes.io/projected/c7885d1c-f834-4a26-9e45-85e170caa048-kube-api-access-k9j4x\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:28.984566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.984428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7885d1c-f834-4a26-9e45-85e170caa048-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:28.984566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.984483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c7885d1c-f834-4a26-9e45-85e170caa048-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:28.984652 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:16:28.984570    2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 22:16:28.984652 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:16:28.984637    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7885d1c-f834-4a26-9e45-85e170caa048-plugin-serving-cert podName:c7885d1c-f834-4a26-9e45-85e170caa048 nodeName:}" failed. No retries permitted until 2026-07-28 22:16:29.484622092 +0000 UTC m=+391.606969962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/c7885d1c-f834-4a26-9e45-85e170caa048-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-bt4hb" (UID: "c7885d1c-f834-4a26-9e45-85e170caa048") : secret "plugin-serving-cert" not found
Jul 28 22:16:28.985060 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.985042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c7885d1c-f834-4a26-9e45-85e170caa048-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:28.993300 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:28.993273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9j4x\" (UniqueName: \"kubernetes.io/projected/c7885d1c-f834-4a26-9e45-85e170caa048-kube-api-access-k9j4x\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:29.259622 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:29.259587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:29.488602 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:29.488562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7885d1c-f834-4a26-9e45-85e170caa048-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:29.491019 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:29.490998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7885d1c-f834-4a26-9e45-85e170caa048-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bt4hb\" (UID: \"c7885d1c-f834-4a26-9e45-85e170caa048\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:29.769750 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:29.769720    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"
Jul 28 22:16:29.777081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:29.777025    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:29.879164 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:29.879131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:29.898967 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:29.898941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb"]
Jul 28 22:16:29.901355 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:16:29.901326    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7885d1c_f834_4a26_9e45_85e170caa048.slice/crio-968bbf5b0427711287777dab74b47f7530efbc44f530bff919e287bbfad206de WatchSource:0}: Error finding container 968bbf5b0427711287777dab74b47f7530efbc44f530bff919e287bbfad206de: Status 404 returned error can't find the container with id 968bbf5b0427711287777dab74b47f7530efbc44f530bff919e287bbfad206de
Jul 28 22:16:29.902743 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:29.902726    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:16:30.336639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:30.336604    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb" event={"ID":"c7885d1c-f834-4a26-9e45-85e170caa048","Type":"ContainerStarted","Data":"968bbf5b0427711287777dab74b47f7530efbc44f530bff919e287bbfad206de"}
Jul 28 22:16:33.201761 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:33.201596    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:33.635686 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:33.635648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:33.736609 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:33.736518    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:37.521625 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:37.521587    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:37.522128 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:37.521654    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:37.528128 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:37.528087    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:38.384272 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.384238    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66c9496995-kx5qc"
Jul 28 22:16:38.469491 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.469456    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"]
Jul 28 22:16:38.490676 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.490646    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"]
Jul 28 22:16:38.490850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.490779    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:38.493582 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.493557    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-smk6t\""
Jul 28 22:16:38.582217 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.582184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcb1b579-29eb-4310-a8a0-c4270812c2d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" (UID: \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:38.582599 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.582262    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hl7\" (UniqueName: \"kubernetes.io/projected/bcb1b579-29eb-4310-a8a0-c4270812c2d5-kube-api-access-z9hl7\") pod \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" (UID: \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:38.683825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.683738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcb1b579-29eb-4310-a8a0-c4270812c2d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" (UID: \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:38.683825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.683799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hl7\" (UniqueName: \"kubernetes.io/projected/bcb1b579-29eb-4310-a8a0-c4270812c2d5-kube-api-access-z9hl7\") pod \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" (UID: \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:38.684203 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.684179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcb1b579-29eb-4310-a8a0-c4270812c2d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" (UID: \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:38.692982 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.692947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hl7\" (UniqueName: \"kubernetes.io/projected/bcb1b579-29eb-4310-a8a0-c4270812c2d5-kube-api-access-z9hl7\") pod \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" (UID: \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:38.803704 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.803662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:38.893614 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.893578    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"]
Jul 28 22:16:38.902892 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.902858    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"]
Jul 28 22:16:38.957073 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.956979    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"]
Jul 28 22:16:38.962506 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.962478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:16:38.980765 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:38.980734    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"]
Jul 28 22:16:39.087682 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:39.087645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szrv\" (UniqueName: \"kubernetes.io/projected/30169899-324e-48a8-9159-489d6dff620d-kube-api-access-7szrv\") pod \"kuadrant-operator-controller-manager-55467d9f7-2nbsj\" (UID: \"30169899-324e-48a8-9159-489d6dff620d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:16:39.087879 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:39.087722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30169899-324e-48a8-9159-489d6dff620d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-2nbsj\" (UID: \"30169899-324e-48a8-9159-489d6dff620d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:16:39.189257 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:39.189221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szrv\" (UniqueName: \"kubernetes.io/projected/30169899-324e-48a8-9159-489d6dff620d-kube-api-access-7szrv\") pod \"kuadrant-operator-controller-manager-55467d9f7-2nbsj\" (UID: \"30169899-324e-48a8-9159-489d6dff620d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:16:39.189447 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:39.189306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30169899-324e-48a8-9159-489d6dff620d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-2nbsj\" (UID: \"30169899-324e-48a8-9159-489d6dff620d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:16:39.189737 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:39.189718    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30169899-324e-48a8-9159-489d6dff620d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-2nbsj\" (UID: \"30169899-324e-48a8-9159-489d6dff620d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:16:39.205996 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:39.205968    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szrv\" (UniqueName: \"kubernetes.io/projected/30169899-324e-48a8-9159-489d6dff620d-kube-api-access-7szrv\") pod \"kuadrant-operator-controller-manager-55467d9f7-2nbsj\" (UID: \"30169899-324e-48a8-9159-489d6dff620d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:16:39.232057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:39.232020    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:39.277247 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:39.277199    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:16:39.330608 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:39.330571    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:45.781313 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:45.781270    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:45.881317 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:45.881280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:48.375881 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:48.375825    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:48.499157 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:48.499115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:51.189812 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:51.189732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:51.266690 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:51.266658    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:51.277837 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:51.277814    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"]
Jul 28 22:16:51.344598 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:16:51.344568    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30169899_324e_48a8_9159_489d6dff620d.slice/crio-68fce465ecd14002dfaf45b153a0c4deed04a31b9a025d4d3e7cc996b7617373 WatchSource:0}: Error finding container 68fce465ecd14002dfaf45b153a0c4deed04a31b9a025d4d3e7cc996b7617373: Status 404 returned error can't find the container with id 68fce465ecd14002dfaf45b153a0c4deed04a31b9a025d4d3e7cc996b7617373
Jul 28 22:16:51.454046 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:51.454007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj" event={"ID":"30169899-324e-48a8-9159-489d6dff620d","Type":"ContainerStarted","Data":"68fce465ecd14002dfaf45b153a0c4deed04a31b9a025d4d3e7cc996b7617373"}
Jul 28 22:16:52.459618 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:52.459584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb" event={"ID":"c7885d1c-f834-4a26-9e45-85e170caa048","Type":"ContainerStarted","Data":"17b2257be7a600ac6bbdd1c4d7f74f3c9247ebc787585461b43d6e2af8ed143b"}
Jul 28 22:16:52.477534 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:52.477491    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bt4hb" podStartSLOduration=2.989363603 podStartE2EDuration="24.477476175s" totalImagesPullingTime="21.488112572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:16:52.475113497 +0000 UTC m=+414.597461384" watchObservedRunningTime="2026-07-28 22:16:52.477476175 +0000 UTC m=+414.599824069"
Jul 28 22:16:53.961094 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:16:53.961028    2560 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-gq7zr_kuadrant-system_bcb1b579-29eb-4310-a8a0-c4270812c2d5_0(50f3ea0e6e4327805291ac55533d9def6faa881394d597a17efcc92cac57b96c): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-gq7zr to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 28 22:16:53.961398 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:16:53.961091    2560 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-gq7zr_kuadrant-system_bcb1b579-29eb-4310-a8a0-c4270812c2d5_0(50f3ea0e6e4327805291ac55533d9def6faa881394d597a17efcc92cac57b96c): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-gq7zr to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:54.469869 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:54.469841    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:54.472253 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:16:54.472212    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" podUID="bcb1b579-29eb-4310-a8a0-c4270812c2d5" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:54.504713 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:54.504681    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:54.507062 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:16:54.507034    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" podUID="bcb1b579-29eb-4310-a8a0-c4270812c2d5" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:54.644257 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:54.644225    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcb1b579-29eb-4310-a8a0-c4270812c2d5-extensions-socket-volume\") pod \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\" (UID: \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\") "
Jul 28 22:16:54.644417 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:54.644337    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9hl7\" (UniqueName: \"kubernetes.io/projected/bcb1b579-29eb-4310-a8a0-c4270812c2d5-kube-api-access-z9hl7\") pod \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\" (UID: \"bcb1b579-29eb-4310-a8a0-c4270812c2d5\") "
Jul 28 22:16:54.644493 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:54.644470    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb1b579-29eb-4310-a8a0-c4270812c2d5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bcb1b579-29eb-4310-a8a0-c4270812c2d5" (UID: "bcb1b579-29eb-4310-a8a0-c4270812c2d5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:16:54.646324 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:54.646302    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb1b579-29eb-4310-a8a0-c4270812c2d5-kube-api-access-z9hl7" (OuterVolumeSpecName: "kube-api-access-z9hl7") pod "bcb1b579-29eb-4310-a8a0-c4270812c2d5" (UID: "bcb1b579-29eb-4310-a8a0-c4270812c2d5"). InnerVolumeSpecName "kube-api-access-z9hl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:16:54.745114 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:54.745017    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9hl7\" (UniqueName: \"kubernetes.io/projected/bcb1b579-29eb-4310-a8a0-c4270812c2d5-kube-api-access-z9hl7\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:16:54.745114 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:54.745050    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcb1b579-29eb-4310-a8a0-c4270812c2d5-extensions-socket-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:16:55.474110 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:55.474069    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:55.476841 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:16:55.476803    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" podUID="bcb1b579-29eb-4310-a8a0-c4270812c2d5" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:55.482126 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:16:55.482074    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gq7zr\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" podUID="bcb1b579-29eb-4310-a8a0-c4270812c2d5" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gq7zr"
Jul 28 22:16:56.408337 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:56.408303    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcb1b579-29eb-4310-a8a0-c4270812c2d5" path="/var/lib/kubelet/pods/bcb1b579-29eb-4310-a8a0-c4270812c2d5/volumes"
Jul 28 22:16:58.177414 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:58.177368    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:58.277538 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:58.277485    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:16:58.490384 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:58.490347    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj" event={"ID":"30169899-324e-48a8-9159-489d6dff620d","Type":"ContainerStarted","Data":"ad78bcca49203ddea6ef3cc041e40ffb0f8944b8ce06236aa3f032415770e373"}
Jul 28 22:16:58.490557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:58.490475    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:16:58.510476 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:16:58.510429    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj" podStartSLOduration=14.364488124 podStartE2EDuration="20.510413836s" totalImagesPullingTime="6.145925712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:16:58.508966862 +0000 UTC m=+420.631314774" watchObservedRunningTime="2026-07-28 22:16:58.510413836 +0000 UTC m=+420.632761729"
Jul 28 22:17:04.587737 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:04.587705    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:04.685957 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:04.685909    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:09.497202 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:09.497176    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:17:21.424920 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.424835    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"]
Jul 28 22:17:21.425473 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.425083    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj" podUID="30169899-324e-48a8-9159-489d6dff620d" containerName="manager" containerID="cri-o://ad78bcca49203ddea6ef3cc041e40ffb0f8944b8ce06236aa3f032415770e373" gracePeriod=10
Jul 28 22:17:21.603240 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.603202    2560 generic.go:356] "Generic (PLEG): container finished" podID="30169899-324e-48a8-9159-489d6dff620d" containerID="ad78bcca49203ddea6ef3cc041e40ffb0f8944b8ce06236aa3f032415770e373" exitCode=0
Jul 28 22:17:21.603375 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.603250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj" event={"ID":"30169899-324e-48a8-9159-489d6dff620d","Type":"ContainerDied","Data":"ad78bcca49203ddea6ef3cc041e40ffb0f8944b8ce06236aa3f032415770e373"}
Jul 28 22:17:21.677491 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.677428    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:17:21.775389 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.775353    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30169899-324e-48a8-9159-489d6dff620d-extensions-socket-volume\") pod \"30169899-324e-48a8-9159-489d6dff620d\" (UID: \"30169899-324e-48a8-9159-489d6dff620d\") "
Jul 28 22:17:21.775582 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.775452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7szrv\" (UniqueName: \"kubernetes.io/projected/30169899-324e-48a8-9159-489d6dff620d-kube-api-access-7szrv\") pod \"30169899-324e-48a8-9159-489d6dff620d\" (UID: \"30169899-324e-48a8-9159-489d6dff620d\") "
Jul 28 22:17:21.775713 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.775690    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30169899-324e-48a8-9159-489d6dff620d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "30169899-324e-48a8-9159-489d6dff620d" (UID: "30169899-324e-48a8-9159-489d6dff620d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:17:21.777456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.777431    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30169899-324e-48a8-9159-489d6dff620d-kube-api-access-7szrv" (OuterVolumeSpecName: "kube-api-access-7szrv") pod "30169899-324e-48a8-9159-489d6dff620d" (UID: "30169899-324e-48a8-9159-489d6dff620d"). InnerVolumeSpecName "kube-api-access-7szrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:17:21.876612 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.876572    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30169899-324e-48a8-9159-489d6dff620d-extensions-socket-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:17:21.876612 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:21.876602    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szrv\" (UniqueName: \"kubernetes.io/projected/30169899-324e-48a8-9159-489d6dff620d-kube-api-access-7szrv\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:17:22.609022 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:22.608989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj" event={"ID":"30169899-324e-48a8-9159-489d6dff620d","Type":"ContainerDied","Data":"68fce465ecd14002dfaf45b153a0c4deed04a31b9a025d4d3e7cc996b7617373"}
Jul 28 22:17:22.609441 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:22.609049    2560 scope.go:117] "RemoveContainer" containerID="ad78bcca49203ddea6ef3cc041e40ffb0f8944b8ce06236aa3f032415770e373"
Jul 28 22:17:22.609441 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:22.608997    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"
Jul 28 22:17:22.627873 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:22.627845    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"]
Jul 28 22:17:22.634161 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:22.634138    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2nbsj"]
Jul 28 22:17:24.406852 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:24.406817    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30169899-324e-48a8-9159-489d6dff620d" path="/var/lib/kubelet/pods/30169899-324e-48a8-9159-489d6dff620d/volumes"
Jul 28 22:17:25.953315 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:25.953280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:26.113241 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:26.113206    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:29.165805 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.165768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:29.172452 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.172421    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-95zlc"]
Jul 28 22:17:29.172814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.172801    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30169899-324e-48a8-9159-489d6dff620d" containerName="manager"
Jul 28 22:17:29.172861 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.172815    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30169899-324e-48a8-9159-489d6dff620d" containerName="manager"
Jul 28 22:17:29.172897 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.172887    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30169899-324e-48a8-9159-489d6dff620d" containerName="manager"
Jul 28 22:17:29.180202 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.180183    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:29.182874 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.182853    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 22:17:29.186017 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.185996    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-95zlc"]
Jul 28 22:17:29.243586 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.243550    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c6202060-5da1-40ee-a03e-8f94c0b862f3-config-file\") pod \"limitador-limitador-64c94fd64f-95zlc\" (UID: \"c6202060-5da1-40ee-a03e-8f94c0b862f3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:29.243764 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.243692    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j58k\" (UniqueName: \"kubernetes.io/projected/c6202060-5da1-40ee-a03e-8f94c0b862f3-kube-api-access-6j58k\") pod \"limitador-limitador-64c94fd64f-95zlc\" (UID: \"c6202060-5da1-40ee-a03e-8f94c0b862f3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:29.281675 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.281643    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-95zlc"]
Jul 28 22:17:29.288404 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.288371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:29.342495 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.342463    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:29.344579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.344557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c6202060-5da1-40ee-a03e-8f94c0b862f3-config-file\") pod \"limitador-limitador-64c94fd64f-95zlc\" (UID: \"c6202060-5da1-40ee-a03e-8f94c0b862f3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:29.344682 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.344639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j58k\" (UniqueName: \"kubernetes.io/projected/c6202060-5da1-40ee-a03e-8f94c0b862f3-kube-api-access-6j58k\") pod \"limitador-limitador-64c94fd64f-95zlc\" (UID: \"c6202060-5da1-40ee-a03e-8f94c0b862f3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:29.345160 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.345145    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c6202060-5da1-40ee-a03e-8f94c0b862f3-config-file\") pod \"limitador-limitador-64c94fd64f-95zlc\" (UID: \"c6202060-5da1-40ee-a03e-8f94c0b862f3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:29.352894 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.352869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j58k\" (UniqueName: \"kubernetes.io/projected/c6202060-5da1-40ee-a03e-8f94c0b862f3-kube-api-access-6j58k\") pod \"limitador-limitador-64c94fd64f-95zlc\" (UID: \"c6202060-5da1-40ee-a03e-8f94c0b862f3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:29.492296 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.492263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:29.629854 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.629831    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-95zlc"]
Jul 28 22:17:29.632248 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:17:29.632223    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6202060_5da1_40ee_a03e_8f94c0b862f3.slice/crio-2d20d887eb4c6dc6ed13cebb21a8389217ce2b7685b0aa9376cb2a44990b67b3 WatchSource:0}: Error finding container 2d20d887eb4c6dc6ed13cebb21a8389217ce2b7685b0aa9376cb2a44990b67b3: Status 404 returned error can't find the container with id 2d20d887eb4c6dc6ed13cebb21a8389217ce2b7685b0aa9376cb2a44990b67b3
Jul 28 22:17:29.642490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.642465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc" event={"ID":"c6202060-5da1-40ee-a03e-8f94c0b862f3","Type":"ContainerStarted","Data":"2d20d887eb4c6dc6ed13cebb21a8389217ce2b7685b0aa9376cb2a44990b67b3"}
Jul 28 22:17:29.978986 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.978954    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-n6rvk"]
Jul 28 22:17:29.983905 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.983882    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n6rvk"
Jul 28 22:17:29.986433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.986408    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6cgks\""
Jul 28 22:17:29.989283 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:29.989262    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-n6rvk"]
Jul 28 22:17:30.007784 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:30.007760    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:30.052085 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:30.052055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbn5f\" (UniqueName: \"kubernetes.io/projected/9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8-kube-api-access-nbn5f\") pod \"authorino-58cd955675-n6rvk\" (UID: \"9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8\") " pod="kuadrant-system/authorino-58cd955675-n6rvk"
Jul 28 22:17:30.071585 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:30.071549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:30.153528 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:30.153493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbn5f\" (UniqueName: \"kubernetes.io/projected/9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8-kube-api-access-nbn5f\") pod \"authorino-58cd955675-n6rvk\" (UID: \"9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8\") " pod="kuadrant-system/authorino-58cd955675-n6rvk"
Jul 28 22:17:30.173629 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:30.172569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbn5f\" (UniqueName: \"kubernetes.io/projected/9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8-kube-api-access-nbn5f\") pod \"authorino-58cd955675-n6rvk\" (UID: \"9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8\") " pod="kuadrant-system/authorino-58cd955675-n6rvk"
Jul 28 22:17:30.294260 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:30.294178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n6rvk"
Jul 28 22:17:30.471529 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:30.471469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-n6rvk"]
Jul 28 22:17:30.475467 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:17:30.475436    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fbcc9ca_7fd1_4a4b_a530_6ea0b627b1b8.slice/crio-9d79becc81ec485fdbfff1d359063915aa044b98bf1e878af186b8ae50b29ddb WatchSource:0}: Error finding container 9d79becc81ec485fdbfff1d359063915aa044b98bf1e878af186b8ae50b29ddb: Status 404 returned error can't find the container with id 9d79becc81ec485fdbfff1d359063915aa044b98bf1e878af186b8ae50b29ddb
Jul 28 22:17:30.649623 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:30.649533    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n6rvk" event={"ID":"9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8","Type":"ContainerStarted","Data":"9d79becc81ec485fdbfff1d359063915aa044b98bf1e878af186b8ae50b29ddb"}
Jul 28 22:17:32.662197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:32.662161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc" event={"ID":"c6202060-5da1-40ee-a03e-8f94c0b862f3","Type":"ContainerStarted","Data":"e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81"}
Jul 28 22:17:32.662597 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:32.662288    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:32.676970 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:32.676922    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc" podStartSLOduration=0.786610528 podStartE2EDuration="3.676903408s" totalImagesPullingTime="2.89029288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:17:32.675636593 +0000 UTC m=+454.797984506" watchObservedRunningTime="2026-07-28 22:17:32.676903408 +0000 UTC m=+454.799251338"
Jul 28 22:17:34.674011 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:34.673975    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n6rvk" event={"ID":"9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8","Type":"ContainerStarted","Data":"501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea"}
Jul 28 22:17:34.690429 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:34.690339    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-n6rvk" podStartSLOduration=1.7396685299999999 podStartE2EDuration="5.690324722s" totalImagesPullingTime="3.950656192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:17:34.688519547 +0000 UTC m=+456.810867440" watchObservedRunningTime="2026-07-28 22:17:34.690324722 +0000 UTC m=+456.812672614"
Jul 28 22:17:35.532132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:35.532080    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-n6rvk"]
Jul 28 22:17:36.682834 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:36.682776    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-n6rvk" podUID="9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8" containerName="authorino" containerID="cri-o://501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea" gracePeriod=30
Jul 28 22:17:36.916418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:36.916392    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n6rvk"
Jul 28 22:17:37.020581 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.020546    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbn5f\" (UniqueName: \"kubernetes.io/projected/9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8-kube-api-access-nbn5f\") pod \"9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8\" (UID: \"9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8\") "
Jul 28 22:17:37.022560 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.022532    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8-kube-api-access-nbn5f" (OuterVolumeSpecName: "kube-api-access-nbn5f") pod "9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8" (UID: "9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8"). InnerVolumeSpecName "kube-api-access-nbn5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:17:37.122202 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.122157    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbn5f\" (UniqueName: \"kubernetes.io/projected/9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8-kube-api-access-nbn5f\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:17:37.689632 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.689599    2560 generic.go:356] "Generic (PLEG): container finished" podID="9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8" containerID="501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea" exitCode=0
Jul 28 22:17:37.690080 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.689642    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n6rvk"
Jul 28 22:17:37.690080 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.689670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n6rvk" event={"ID":"9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8","Type":"ContainerDied","Data":"501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea"}
Jul 28 22:17:37.690080 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.689702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n6rvk" event={"ID":"9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8","Type":"ContainerDied","Data":"9d79becc81ec485fdbfff1d359063915aa044b98bf1e878af186b8ae50b29ddb"}
Jul 28 22:17:37.690080 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.689719    2560 scope.go:117] "RemoveContainer" containerID="501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea"
Jul 28 22:17:37.698231 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.698213    2560 scope.go:117] "RemoveContainer" containerID="501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea"
Jul 28 22:17:37.698474 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:17:37.698455    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea\": container with ID starting with 501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea not found: ID does not exist" containerID="501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea"
Jul 28 22:17:37.698522 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.698484    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea"} err="failed to get container status \"501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea\": rpc error: code = NotFound desc = could not find container \"501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea\": container with ID starting with 501bde82c8fc26b910aa50a0b84c1af23126a8804c74bea11cd0bd0e5b5cb5ea not found: ID does not exist"
Jul 28 22:17:37.710165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.710142    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-n6rvk"]
Jul 28 22:17:37.713613 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:37.713591    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-n6rvk"]
Jul 28 22:17:38.407231 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:38.407200    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8" path="/var/lib/kubelet/pods/9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8/volumes"
Jul 28 22:17:43.669065 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:43.669031    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:44.524086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:44.524055    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-95zlc"]
Jul 28 22:17:44.524308 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:44.524261    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc" podUID="c6202060-5da1-40ee-a03e-8f94c0b862f3" containerName="limitador" containerID="cri-o://e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81" gracePeriod=30
Jul 28 22:17:45.066955 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.066932    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:45.194619 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.194531    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c6202060-5da1-40ee-a03e-8f94c0b862f3-config-file\") pod \"c6202060-5da1-40ee-a03e-8f94c0b862f3\" (UID: \"c6202060-5da1-40ee-a03e-8f94c0b862f3\") "
Jul 28 22:17:45.194619 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.194572    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j58k\" (UniqueName: \"kubernetes.io/projected/c6202060-5da1-40ee-a03e-8f94c0b862f3-kube-api-access-6j58k\") pod \"c6202060-5da1-40ee-a03e-8f94c0b862f3\" (UID: \"c6202060-5da1-40ee-a03e-8f94c0b862f3\") "
Jul 28 22:17:45.194910 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.194888    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6202060-5da1-40ee-a03e-8f94c0b862f3-config-file" (OuterVolumeSpecName: "config-file") pod "c6202060-5da1-40ee-a03e-8f94c0b862f3" (UID: "c6202060-5da1-40ee-a03e-8f94c0b862f3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:17:45.196637 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.196612    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6202060-5da1-40ee-a03e-8f94c0b862f3-kube-api-access-6j58k" (OuterVolumeSpecName: "kube-api-access-6j58k") pod "c6202060-5da1-40ee-a03e-8f94c0b862f3" (UID: "c6202060-5da1-40ee-a03e-8f94c0b862f3"). InnerVolumeSpecName "kube-api-access-6j58k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:17:45.295973 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.295932    2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c6202060-5da1-40ee-a03e-8f94c0b862f3-config-file\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:17:45.295973 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.295964    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j58k\" (UniqueName: \"kubernetes.io/projected/c6202060-5da1-40ee-a03e-8f94c0b862f3-kube-api-access-6j58k\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:17:45.734367 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.734328    2560 generic.go:356] "Generic (PLEG): container finished" podID="c6202060-5da1-40ee-a03e-8f94c0b862f3" containerID="e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81" exitCode=0
Jul 28 22:17:45.734539 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.734386    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc"
Jul 28 22:17:45.734539 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.734412    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc" event={"ID":"c6202060-5da1-40ee-a03e-8f94c0b862f3","Type":"ContainerDied","Data":"e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81"}
Jul 28 22:17:45.734539 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.734487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-95zlc" event={"ID":"c6202060-5da1-40ee-a03e-8f94c0b862f3","Type":"ContainerDied","Data":"2d20d887eb4c6dc6ed13cebb21a8389217ce2b7685b0aa9376cb2a44990b67b3"}
Jul 28 22:17:45.734539 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.734509    2560 scope.go:117] "RemoveContainer" containerID="e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81"
Jul 28 22:17:45.743027 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.743013    2560 scope.go:117] "RemoveContainer" containerID="e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81"
Jul 28 22:17:45.743267 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:17:45.743251    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81\": container with ID starting with e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81 not found: ID does not exist" containerID="e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81"
Jul 28 22:17:45.743312 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.743275    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81"} err="failed to get container status \"e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81\": rpc error: code = NotFound desc = could not find container \"e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81\": container with ID starting with e0bcb8d6803534327aa6ffdb20de595d9a4be65eda079c7480b80200a7ee6f81 not found: ID does not exist"
Jul 28 22:17:45.754626 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.754599    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-95zlc"]
Jul 28 22:17:45.756677 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:45.756660    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-95zlc"]
Jul 28 22:17:46.406614 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:46.406575    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6202060-5da1-40ee-a03e-8f94c0b862f3" path="/var/lib/kubelet/pods/c6202060-5da1-40ee-a03e-8f94c0b862f3/volumes"
Jul 28 22:17:48.957116 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:48.957073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:49.059147 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.059095    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:49.415283 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.415254    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"]
Jul 28 22:17:49.415622 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.415610    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6202060-5da1-40ee-a03e-8f94c0b862f3" containerName="limitador"
Jul 28 22:17:49.415662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.415624    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6202060-5da1-40ee-a03e-8f94c0b862f3" containerName="limitador"
Jul 28 22:17:49.415662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.415645    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8" containerName="authorino"
Jul 28 22:17:49.415662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.415651    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8" containerName="authorino"
Jul 28 22:17:49.415752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.415709    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbcc9ca-7fd1-4a4b-a530-6ea0b627b1b8" containerName="authorino"
Jul 28 22:17:49.415752 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.415716    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6202060-5da1-40ee-a03e-8f94c0b862f3" containerName="limitador"
Jul 28 22:17:49.418894 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.418875    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:17:49.422530 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.422511    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 22:17:49.423427 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.423410    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 22:17:49.423547 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.423410    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 28 22:17:49.423547 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.423412    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zt9dz\""
Jul 28 22:17:49.430022 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.430000    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"]
Jul 28 22:17:49.513606 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.513573    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:17:49.537621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.537587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlqbz\" (UniqueName: \"kubernetes.io/projected/8fd46e63-d882-4945-bc82-cdce94366ed2-kube-api-access-xlqbz\") pod \"postgres-69fb4594cf-wknnf\" (UID: \"8fd46e63-d882-4945-bc82-cdce94366ed2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:17:49.537830 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.537655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8fd46e63-d882-4945-bc82-cdce94366ed2-data\") pod \"postgres-69fb4594cf-wknnf\" (UID: \"8fd46e63-d882-4945-bc82-cdce94366ed2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:17:49.638820 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.638783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8fd46e63-d882-4945-bc82-cdce94366ed2-data\") pod \"postgres-69fb4594cf-wknnf\" (UID: \"8fd46e63-d882-4945-bc82-cdce94366ed2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:17:49.639003 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.638898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlqbz\" (UniqueName: \"kubernetes.io/projected/8fd46e63-d882-4945-bc82-cdce94366ed2-kube-api-access-xlqbz\") pod \"postgres-69fb4594cf-wknnf\" (UID: \"8fd46e63-d882-4945-bc82-cdce94366ed2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:17:49.639189 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.639170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8fd46e63-d882-4945-bc82-cdce94366ed2-data\") pod \"postgres-69fb4594cf-wknnf\" (UID: \"8fd46e63-d882-4945-bc82-cdce94366ed2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:17:49.647252 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.647231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlqbz\" (UniqueName: \"kubernetes.io/projected/8fd46e63-d882-4945-bc82-cdce94366ed2-kube-api-access-xlqbz\") pod \"postgres-69fb4594cf-wknnf\" (UID: \"8fd46e63-d882-4945-bc82-cdce94366ed2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:17:49.732382 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.732357    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:17:49.860616 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:49.860591    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"]
Jul 28 22:17:49.863002 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:17:49.862976    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd46e63_d882_4945_bc82_cdce94366ed2.slice/crio-be97a9b376662f917e33e73061276a1fb6ac9051d0eb777f2a3eab8bd0050da8 WatchSource:0}: Error finding container be97a9b376662f917e33e73061276a1fb6ac9051d0eb777f2a3eab8bd0050da8: Status 404 returned error can't find the container with id be97a9b376662f917e33e73061276a1fb6ac9051d0eb777f2a3eab8bd0050da8
Jul 28 22:17:50.764524 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:50.764450    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf" event={"ID":"8fd46e63-d882-4945-bc82-cdce94366ed2","Type":"ContainerStarted","Data":"be97a9b376662f917e33e73061276a1fb6ac9051d0eb777f2a3eab8bd0050da8"}
Jul 28 22:17:54.794727 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:54.794682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf" event={"ID":"8fd46e63-d882-4945-bc82-cdce94366ed2","Type":"ContainerStarted","Data":"4316afc97d761648e326cde565bd62f13699a921fd3699f8e6034122485163f4"}
Jul 28 22:17:54.795190 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:54.794756    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:17:54.812940 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:54.812877    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf" podStartSLOduration=0.974270441 podStartE2EDuration="5.812861272s" totalImagesPullingTime="4.838590831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:17:54.811589731 +0000 UTC m=+476.933937643" watchObservedRunningTime="2026-07-28 22:17:54.812861272 +0000 UTC m=+476.935209164"
Jul 28 22:17:58.273960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:17:58.273929    2560 scope.go:117] "RemoveContainer" containerID="7908b3c0f25ec7b57c3dd4ee32afa89ae0c398c4e9678301bc4085619619cfd6"
Jul 28 22:18:00.827854 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:00.827826    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wknnf"
Jul 28 22:18:01.138938 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.138843    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:01.363911 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.363880    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wm56j"]
Jul 28 22:18:01.370266 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.370249    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wm56j"
Jul 28 22:18:01.372705 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.372676    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6cgks\""
Jul 28 22:18:01.373482 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.373459    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wm56j"]
Jul 28 22:18:01.446255 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.446172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnndh\" (UniqueName: \"kubernetes.io/projected/b84602bb-39dc-4339-806a-287e23a3d02b-kube-api-access-qnndh\") pod \"authorino-5d4b67c446-wm56j\" (UID: \"b84602bb-39dc-4339-806a-287e23a3d02b\") " pod="kuadrant-system/authorino-5d4b67c446-wm56j"
Jul 28 22:18:01.476478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.476446    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:01.546881 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.546849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnndh\" (UniqueName: \"kubernetes.io/projected/b84602bb-39dc-4339-806a-287e23a3d02b-kube-api-access-qnndh\") pod \"authorino-5d4b67c446-wm56j\" (UID: \"b84602bb-39dc-4339-806a-287e23a3d02b\") " pod="kuadrant-system/authorino-5d4b67c446-wm56j"
Jul 28 22:18:01.555290 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.555259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnndh\" (UniqueName: \"kubernetes.io/projected/b84602bb-39dc-4339-806a-287e23a3d02b-kube-api-access-qnndh\") pod \"authorino-5d4b67c446-wm56j\" (UID: \"b84602bb-39dc-4339-806a-287e23a3d02b\") " pod="kuadrant-system/authorino-5d4b67c446-wm56j"
Jul 28 22:18:01.598795 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.598761    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wm56j"]
Jul 28 22:18:01.599046 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.599034    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wm56j"
Jul 28 22:18:01.622262 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.622233    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68d87b87c4-ldkx4"]
Jul 28 22:18:01.627850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.627689    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d87b87c4-ldkx4"
Jul 28 22:18:01.631870 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.631844    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68d87b87c4-ldkx4"]
Jul 28 22:18:01.736408 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.736383    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wm56j"]
Jul 28 22:18:01.738490 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:18:01.738462    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb84602bb_39dc_4339_806a_287e23a3d02b.slice/crio-7faf55137d5dfe5ba9544092d11701b9e207689be5d11dbcf4fddd1858a8c50a WatchSource:0}: Error finding container 7faf55137d5dfe5ba9544092d11701b9e207689be5d11dbcf4fddd1858a8c50a: Status 404 returned error can't find the container with id 7faf55137d5dfe5ba9544092d11701b9e207689be5d11dbcf4fddd1858a8c50a
Jul 28 22:18:01.748395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.748368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9gp\" (UniqueName: \"kubernetes.io/projected/ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93-kube-api-access-nj9gp\") pod \"authorino-68d87b87c4-ldkx4\" (UID: \"ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93\") " pod="kuadrant-system/authorino-68d87b87c4-ldkx4"
Jul 28 22:18:01.758815 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.758786    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68d87b87c4-ldkx4"]
Jul 28 22:18:01.759082 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:18:01.759057    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nj9gp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-68d87b87c4-ldkx4" podUID="ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93"
Jul 28 22:18:01.786405 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.786379    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-549599695d-kdq7j"]
Jul 28 22:18:01.790154 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.790138    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:18:01.792644 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.792625    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 22:18:01.797320 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.797299    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-549599695d-kdq7j"]
Jul 28 22:18:01.831293 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.831265    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d87b87c4-ldkx4"
Jul 28 22:18:01.831293 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.831282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wm56j" event={"ID":"b84602bb-39dc-4339-806a-287e23a3d02b","Type":"ContainerStarted","Data":"7faf55137d5dfe5ba9544092d11701b9e207689be5d11dbcf4fddd1858a8c50a"}
Jul 28 22:18:01.837970 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.837951    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d87b87c4-ldkx4"
Jul 28 22:18:01.849731 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.849708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-tls-cert\") pod \"authorino-549599695d-kdq7j\" (UID: \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\") " pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:18:01.849857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.849745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9gp\" (UniqueName: \"kubernetes.io/projected/ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93-kube-api-access-nj9gp\") pod \"authorino-68d87b87c4-ldkx4\" (UID: \"ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93\") " pod="kuadrant-system/authorino-68d87b87c4-ldkx4"
Jul 28 22:18:01.849857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.849795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8cdd\" (UniqueName: \"kubernetes.io/projected/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-kube-api-access-h8cdd\") pod \"authorino-549599695d-kdq7j\" (UID: \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\") " pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:18:01.858056 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.858037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9gp\" (UniqueName: \"kubernetes.io/projected/ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93-kube-api-access-nj9gp\") pod \"authorino-68d87b87c4-ldkx4\" (UID: \"ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93\") " pod="kuadrant-system/authorino-68d87b87c4-ldkx4"
Jul 28 22:18:01.950697 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.950660    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj9gp\" (UniqueName: \"kubernetes.io/projected/ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93-kube-api-access-nj9gp\") pod \"ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93\" (UID: \"ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93\") "
Jul 28 22:18:01.950867 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.950775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cdd\" (UniqueName: \"kubernetes.io/projected/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-kube-api-access-h8cdd\") pod \"authorino-549599695d-kdq7j\" (UID: \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\") " pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:18:01.950867 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.950848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-tls-cert\") pod \"authorino-549599695d-kdq7j\" (UID: \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\") " pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:18:01.952799 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.952769    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93-kube-api-access-nj9gp" (OuterVolumeSpecName: "kube-api-access-nj9gp") pod "ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93" (UID: "ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93"). InnerVolumeSpecName "kube-api-access-nj9gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:18:01.953164 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.953147    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-tls-cert\") pod \"authorino-549599695d-kdq7j\" (UID: \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\") " pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:18:01.958212 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:01.958193    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cdd\" (UniqueName: \"kubernetes.io/projected/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-kube-api-access-h8cdd\") pod \"authorino-549599695d-kdq7j\" (UID: \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\") " pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:18:02.051326 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.051307    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9gp\" (UniqueName: \"kubernetes.io/projected/ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93-kube-api-access-nj9gp\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:18:02.100737 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.100716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:18:02.235292 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.235208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-549599695d-kdq7j"]
Jul 28 22:18:02.237460 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:18:02.237437    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbecf44c_45e0_4e3e_9619_eb9a0afabfd4.slice/crio-05da200e8f7e1d85a6c916077fa726923111330896e33f044212e7215102e9d9 WatchSource:0}: Error finding container 05da200e8f7e1d85a6c916077fa726923111330896e33f044212e7215102e9d9: Status 404 returned error can't find the container with id 05da200e8f7e1d85a6c916077fa726923111330896e33f044212e7215102e9d9
Jul 28 22:18:02.837763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.837727    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wm56j" event={"ID":"b84602bb-39dc-4339-806a-287e23a3d02b","Type":"ContainerStarted","Data":"8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5"}
Jul 28 22:18:02.837763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.837740    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-wm56j" podUID="b84602bb-39dc-4339-806a-287e23a3d02b" containerName="authorino" containerID="cri-o://8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5" gracePeriod=30
Jul 28 22:18:02.839306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.839283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d87b87c4-ldkx4"
Jul 28 22:18:02.839433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.839311    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-549599695d-kdq7j" event={"ID":"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4","Type":"ContainerStarted","Data":"83340c0afabec052e36bf0b6c62f9ead47123e6f868db6da7bfb82477ac0958d"}
Jul 28 22:18:02.839433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.839348    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-549599695d-kdq7j" event={"ID":"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4","Type":"ContainerStarted","Data":"05da200e8f7e1d85a6c916077fa726923111330896e33f044212e7215102e9d9"}
Jul 28 22:18:02.853006 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.852963    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-wm56j" podStartSLOduration=1.543757888 podStartE2EDuration="1.852948669s" totalImagesPullingTime="309.190781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:18:02.852449511 +0000 UTC m=+484.974797405" watchObservedRunningTime="2026-07-28 22:18:02.852948669 +0000 UTC m=+484.975296562"
Jul 28 22:18:02.874412 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.874383    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68d87b87c4-ldkx4"]
Jul 28 22:18:02.879403 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.879382    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68d87b87c4-ldkx4"]
Jul 28 22:18:02.893282 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:02.893228    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-549599695d-kdq7j" podStartSLOduration=1.553729649 podStartE2EDuration="1.893206115s" totalImagesPullingTime="339.476466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:18:02.891715123 +0000 UTC m=+485.014063012" watchObservedRunningTime="2026-07-28 22:18:02.893206115 +0000 UTC m=+485.015554009"
Jul 28 22:18:03.075866 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.075841    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wm56j"
Jul 28 22:18:03.161287 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.161182    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnndh\" (UniqueName: \"kubernetes.io/projected/b84602bb-39dc-4339-806a-287e23a3d02b-kube-api-access-qnndh\") pod \"b84602bb-39dc-4339-806a-287e23a3d02b\" (UID: \"b84602bb-39dc-4339-806a-287e23a3d02b\") "
Jul 28 22:18:03.163225 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.163203    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84602bb-39dc-4339-806a-287e23a3d02b-kube-api-access-qnndh" (OuterVolumeSpecName: "kube-api-access-qnndh") pod "b84602bb-39dc-4339-806a-287e23a3d02b" (UID: "b84602bb-39dc-4339-806a-287e23a3d02b"). InnerVolumeSpecName "kube-api-access-qnndh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:18:03.262219 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.262179    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnndh\" (UniqueName: \"kubernetes.io/projected/b84602bb-39dc-4339-806a-287e23a3d02b-kube-api-access-qnndh\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:18:03.845343 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.845261    2560 generic.go:356] "Generic (PLEG): container finished" podID="b84602bb-39dc-4339-806a-287e23a3d02b" containerID="8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5" exitCode=0
Jul 28 22:18:03.845343 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.845306    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wm56j"
Jul 28 22:18:03.845802 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.845350    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wm56j" event={"ID":"b84602bb-39dc-4339-806a-287e23a3d02b","Type":"ContainerDied","Data":"8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5"}
Jul 28 22:18:03.845802 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.845395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wm56j" event={"ID":"b84602bb-39dc-4339-806a-287e23a3d02b","Type":"ContainerDied","Data":"7faf55137d5dfe5ba9544092d11701b9e207689be5d11dbcf4fddd1858a8c50a"}
Jul 28 22:18:03.845802 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.845416    2560 scope.go:117] "RemoveContainer" containerID="8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5"
Jul 28 22:18:03.853904 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.853888    2560 scope.go:117] "RemoveContainer" containerID="8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5"
Jul 28 22:18:03.854185 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:18:03.854166    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5\": container with ID starting with 8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5 not found: ID does not exist" containerID="8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5"
Jul 28 22:18:03.854264 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.854193    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5"} err="failed to get container status \"8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5\": rpc error: code = NotFound desc = could not find container \"8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5\": container with ID starting with 8af5c3307aab09f02651b0cd9cbb2a0c08993f603c651ac8b2ce82532a941ac5 not found: ID does not exist"
Jul 28 22:18:03.866250 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.866227    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wm56j"]
Jul 28 22:18:03.869806 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:03.869786    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wm56j"]
Jul 28 22:18:04.406708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:04.406667    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93" path="/var/lib/kubelet/pods/ab3ce85d-552e-4bb6-95b0-db2c8cfe8c93/volumes"
Jul 28 22:18:04.406920 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:04.406908    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b84602bb-39dc-4339-806a-287e23a3d02b" path="/var/lib/kubelet/pods/b84602bb-39dc-4339-806a-287e23a3d02b/volumes"
Jul 28 22:18:07.590401 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:07.590362    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:07.680264 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:07.680231    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:07.753416 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:07.753381    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:08.372438 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:08.372392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:11.462432 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:11.462393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:11.562582 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:11.562548    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:43.059248 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:43.059209    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:43.160984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:43.160948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:43.697048 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:43.696984    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:44.373799 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.373765    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"]
Jul 28 22:18:44.374359 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.374339    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b84602bb-39dc-4339-806a-287e23a3d02b" containerName="authorino"
Jul 28 22:18:44.374443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.374362    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b84602bb-39dc-4339-806a-287e23a3d02b" containerName="authorino"
Jul 28 22:18:44.374501 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.374489    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b84602bb-39dc-4339-806a-287e23a3d02b" containerName="authorino"
Jul 28 22:18:44.377528 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.377504    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:44.380092 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.380073    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 22:18:44.388539 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.388515    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"]
Jul 28 22:18:44.444734 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.444693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjshg\" (UniqueName: \"kubernetes.io/projected/14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8-kube-api-access-mjshg\") pod \"payload-pre-processing-687dbb6cf6-9jqb6\" (UID: \"14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:44.444902 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.444767    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-9jqb6\" (UID: \"14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:44.508621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.508587    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-7g7wm"]
Jul 28 22:18:44.511853 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.511835    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:18:44.514515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.514494    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vh2bw\""
Jul 28 22:18:44.523075 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.523045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-7g7wm"]
Jul 28 22:18:44.545664 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.545630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjshg\" (UniqueName: \"kubernetes.io/projected/14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8-kube-api-access-mjshg\") pod \"payload-pre-processing-687dbb6cf6-9jqb6\" (UID: \"14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:44.545825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.545691    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5v6\" (UniqueName: \"kubernetes.io/projected/3e34ca30-b28c-4cca-8472-d23bcf1ddf51-kube-api-access-kt5v6\") pod \"payload-processing-5cc6cd444b-7g7wm\" (UID: \"3e34ca30-b28c-4cca-8472-d23bcf1ddf51\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:18:44.545825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.545768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-9jqb6\" (UID: \"14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:44.545825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.545796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e34ca30-b28c-4cca-8472-d23bcf1ddf51-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-7g7wm\" (UID: \"3e34ca30-b28c-4cca-8472-d23bcf1ddf51\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:18:44.546477 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.546460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-9jqb6\" (UID: \"14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:44.554343 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.554315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjshg\" (UniqueName: \"kubernetes.io/projected/14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8-kube-api-access-mjshg\") pod \"payload-pre-processing-687dbb6cf6-9jqb6\" (UID: \"14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:44.646719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.646622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5v6\" (UniqueName: \"kubernetes.io/projected/3e34ca30-b28c-4cca-8472-d23bcf1ddf51-kube-api-access-kt5v6\") pod \"payload-processing-5cc6cd444b-7g7wm\" (UID: \"3e34ca30-b28c-4cca-8472-d23bcf1ddf51\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:18:44.646719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.646669    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e34ca30-b28c-4cca-8472-d23bcf1ddf51-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-7g7wm\" (UID: \"3e34ca30-b28c-4cca-8472-d23bcf1ddf51\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:18:44.647392 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.647373    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e34ca30-b28c-4cca-8472-d23bcf1ddf51-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-7g7wm\" (UID: \"3e34ca30-b28c-4cca-8472-d23bcf1ddf51\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:18:44.654427 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.654403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5v6\" (UniqueName: \"kubernetes.io/projected/3e34ca30-b28c-4cca-8472-d23bcf1ddf51-kube-api-access-kt5v6\") pod \"payload-processing-5cc6cd444b-7g7wm\" (UID: \"3e34ca30-b28c-4cca-8472-d23bcf1ddf51\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:18:44.688412 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.688370    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:44.821549 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.821521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"]
Jul 28 22:18:44.823508 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.823479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:18:44.824140 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:18:44.824116    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e0eb05_7d4b_4f4b_8c01_5b671e6d1fb8.slice/crio-42da15a27ed3138a1832e8a36831fbf5474637d8301c1570e4652d74e59ec85f WatchSource:0}: Error finding container 42da15a27ed3138a1832e8a36831fbf5474637d8301c1570e4652d74e59ec85f: Status 404 returned error can't find the container with id 42da15a27ed3138a1832e8a36831fbf5474637d8301c1570e4652d74e59ec85f
Jul 28 22:18:44.957321 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:44.957298    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-7g7wm"]
Jul 28 22:18:44.959135 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:18:44.959110    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e34ca30_b28c_4cca_8472_d23bcf1ddf51.slice/crio-0811ec79415687e11e17d4c7110ffc74fe53ff13c629bff74dca6557c52666df WatchSource:0}: Error finding container 0811ec79415687e11e17d4c7110ffc74fe53ff13c629bff74dca6557c52666df: Status 404 returned error can't find the container with id 0811ec79415687e11e17d4c7110ffc74fe53ff13c629bff74dca6557c52666df
Jul 28 22:18:45.047477 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:45.047440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm" event={"ID":"3e34ca30-b28c-4cca-8472-d23bcf1ddf51","Type":"ContainerStarted","Data":"0811ec79415687e11e17d4c7110ffc74fe53ff13c629bff74dca6557c52666df"}
Jul 28 22:18:45.048514 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:45.048491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6" event={"ID":"14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8","Type":"ContainerStarted","Data":"42da15a27ed3138a1832e8a36831fbf5474637d8301c1570e4652d74e59ec85f"}
Jul 28 22:18:45.091987 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:45.091938    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:45.194894 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:45.194807    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:47.063389 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:47.063352    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm" event={"ID":"3e34ca30-b28c-4cca-8472-d23bcf1ddf51","Type":"ContainerStarted","Data":"cd608bd926a3cc9657ae453985045f82d6313b144debbf6d40e19d6761b4dfca"}
Jul 28 22:18:47.063856 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:47.063443    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:18:47.064919 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:47.064892    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6" event={"ID":"14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8","Type":"ContainerStarted","Data":"a88cf90f3c8fb9e7ed525f6883c58c100a0807cec0a188d93f4069f583c5aa23"}
Jul 28 22:18:47.065051 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:47.064938    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:47.080317 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:47.080265    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm" podStartSLOduration=1.5339839020000001 podStartE2EDuration="3.080248655s" totalImagesPullingTime="1.546264753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:18:47.07903031 +0000 UTC m=+529.201378221" watchObservedRunningTime="2026-07-28 22:18:47.080248655 +0000 UTC m=+529.202596548"
Jul 28 22:18:47.095427 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:47.095372    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6" podStartSLOduration=1.420770304 podStartE2EDuration="3.095354421s" totalImagesPullingTime="1.674584117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:18:47.094391469 +0000 UTC m=+529.216739386" watchObservedRunningTime="2026-07-28 22:18:47.095354421 +0000 UTC m=+529.217702315"
Jul 28 22:18:54.712080 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:54.712000    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:54.813894 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:54.813859    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:18:58.072274 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:58.072243    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-9jqb6"
Jul 28 22:18:58.072742 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:18:58.072325    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-7g7wm"
Jul 28 22:19:03.720807 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:03.720766    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:19:11.768732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.768695    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"]
Jul 28 22:19:11.773408 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.773385    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:11.775962 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.775938    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 28 22:19:11.777845 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.777426    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 22:19:11.777845 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.777439    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-72d9n\""
Jul 28 22:19:11.777845 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.777451    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 22:19:11.781002 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.780965    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"]
Jul 28 22:19:11.804781 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.804744    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:19:11.907445 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.907409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:11.907445 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.907447    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:11.907782 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.907495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:11.907782 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.907515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:11.907782 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.907581    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:11.907782 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:11.907661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7lh\" (UniqueName: \"kubernetes.io/projected/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-kube-api-access-zc7lh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.008400 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.008363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.008400 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.008400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.008633 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.008435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.008633 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.008452    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.008633 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.008482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.008633 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.008530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7lh\" (UniqueName: \"kubernetes.io/projected/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-kube-api-access-zc7lh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.008915 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.008893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.008979 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.008926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.009033 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.008979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.010908 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.010877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.011194 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.011176    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.016696 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.016668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7lh\" (UniqueName: \"kubernetes.io/projected/63fc61f1-aed9-4494-836e-9eabe0b6d6cd-kube-api-access-zc7lh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6mrj\" (UID: \"63fc61f1-aed9-4494-836e-9eabe0b6d6cd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.085895 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.085815    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:12.224695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:12.224669    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"]
Jul 28 22:19:12.227380 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:19:12.227350    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fc61f1_aed9_4494_836e_9eabe0b6d6cd.slice/crio-34224936fe5fe3274b1cac67accf3ec158b3417c52e6fb6d6e933d9b217c4713 WatchSource:0}: Error finding container 34224936fe5fe3274b1cac67accf3ec158b3417c52e6fb6d6e933d9b217c4713: Status 404 returned error can't find the container with id 34224936fe5fe3274b1cac67accf3ec158b3417c52e6fb6d6e933d9b217c4713
Jul 28 22:19:13.197908 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:13.197866    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj" event={"ID":"63fc61f1-aed9-4494-836e-9eabe0b6d6cd","Type":"ContainerStarted","Data":"34224936fe5fe3274b1cac67accf3ec158b3417c52e6fb6d6e933d9b217c4713"}
Jul 28 22:19:14.203921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:14.203876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj" event={"ID":"63fc61f1-aed9-4494-836e-9eabe0b6d6cd","Type":"ContainerStarted","Data":"4518d415b68997a0035a9e9a20572f10297cc0dd307feca230d08ef1a4a83ef9"}
Jul 28 22:19:14.220911 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:14.220863    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj" podStartSLOduration=2.103475712 podStartE2EDuration="3.22084676s" totalImagesPullingTime="1.117371048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:19:14.220416737 +0000 UTC m=+556.342764659" watchObservedRunningTime="2026-07-28 22:19:14.22084676 +0000 UTC m=+556.343194653"
Jul 28 22:19:22.086969 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:22.086928    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:22.087406 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:22.087024    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:22.100285 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:22.100251    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:22.252957 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:22.252930    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6mrj"
Jul 28 22:19:25.363157 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.363126    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"]
Jul 28 22:19:25.369526 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.369496    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.372912 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.372891    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 28 22:19:25.377620 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.377259    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"]
Jul 28 22:19:25.394172 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.394146    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:19:25.436719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.436688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.436861 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.436724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.436861 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.436767    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.436861 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.436789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.436861 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.436812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnm75\" (UniqueName: \"kubernetes.io/projected/0577729c-705f-46c3-8f0d-dbaa92cd51f2-kube-api-access-rnm75\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.437003 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.436864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0577729c-705f-46c3-8f0d-dbaa92cd51f2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.538008 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.537974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.538197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.538018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.538197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.538071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.538197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.538111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.538197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.538146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnm75\" (UniqueName: \"kubernetes.io/projected/0577729c-705f-46c3-8f0d-dbaa92cd51f2-kube-api-access-rnm75\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.538197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.538172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0577729c-705f-46c3-8f0d-dbaa92cd51f2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.538548 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.538509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.538666 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.538576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.538666 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.538606    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.540293 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.540272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0577729c-705f-46c3-8f0d-dbaa92cd51f2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.540609 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.540593    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0577729c-705f-46c3-8f0d-dbaa92cd51f2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.545899 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.545879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnm75\" (UniqueName: \"kubernetes.io/projected/0577729c-705f-46c3-8f0d-dbaa92cd51f2-kube-api-access-rnm75\") pod \"e2e-trlp-test-simulated-kserve-598466548d-drxck\" (UID: \"0577729c-705f-46c3-8f0d-dbaa92cd51f2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.681122 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.681025    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:25.813355 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:25.813330    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"]
Jul 28 22:19:25.815725 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:19:25.815700    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0577729c_705f_46c3_8f0d_dbaa92cd51f2.slice/crio-90c5b084e15cb01af8e031f5b18492fff3340e30aa381a6b91a767bf1e83987b WatchSource:0}: Error finding container 90c5b084e15cb01af8e031f5b18492fff3340e30aa381a6b91a767bf1e83987b: Status 404 returned error can't find the container with id 90c5b084e15cb01af8e031f5b18492fff3340e30aa381a6b91a767bf1e83987b
Jul 28 22:19:26.263194 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:26.263084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck" event={"ID":"0577729c-705f-46c3-8f0d-dbaa92cd51f2","Type":"ContainerStarted","Data":"15922abdcf92a0cab2f661eace4a380699e1bd7e58d5a330d6f90bd965dee72d"}
Jul 28 22:19:26.263194 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:26.263170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck" event={"ID":"0577729c-705f-46c3-8f0d-dbaa92cd51f2","Type":"ContainerStarted","Data":"90c5b084e15cb01af8e031f5b18492fff3340e30aa381a6b91a767bf1e83987b"}
Jul 28 22:19:26.281717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:26.281667    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck" podStartSLOduration=1.090805649 podStartE2EDuration="1.28165132s" totalImagesPullingTime="190.845671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:19:26.279155325 +0000 UTC m=+568.401503221" watchObservedRunningTime="2026-07-28 22:19:26.28165132 +0000 UTC m=+568.403999213"
Jul 28 22:19:32.694299 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:32.694257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:19:35.681785 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:35.681751    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:35.682188 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:35.681825    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:35.693991 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:35.693967    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:36.329991 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:36.329966    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-drxck"
Jul 28 22:19:43.777812 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:43.777741    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:19:51.564517 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:19:51.564475    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:20:58.935742 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:20:58.935707    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:20:59.001023 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:20:59.000982    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:21:00.046535 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.046501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:21:00.144907 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.144867    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:21:00.472063 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.472030    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-549599695d-kdq7j"]
Jul 28 22:21:00.472296 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.472267    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-549599695d-kdq7j" podUID="dbecf44c-45e0-4e3e-9619-eb9a0afabfd4" containerName="authorino" containerID="cri-o://83340c0afabec052e36bf0b6c62f9ead47123e6f868db6da7bfb82477ac0958d" gracePeriod=30
Jul 28 22:21:00.721364 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.721336    2560 generic.go:356] "Generic (PLEG): container finished" podID="dbecf44c-45e0-4e3e-9619-eb9a0afabfd4" containerID="83340c0afabec052e36bf0b6c62f9ead47123e6f868db6da7bfb82477ac0958d" exitCode=0
Jul 28 22:21:00.721515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.721409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-549599695d-kdq7j" event={"ID":"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4","Type":"ContainerDied","Data":"83340c0afabec052e36bf0b6c62f9ead47123e6f868db6da7bfb82477ac0958d"}
Jul 28 22:21:00.721515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.721455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-549599695d-kdq7j" event={"ID":"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4","Type":"ContainerDied","Data":"05da200e8f7e1d85a6c916077fa726923111330896e33f044212e7215102e9d9"}
Jul 28 22:21:00.721515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.721471    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05da200e8f7e1d85a6c916077fa726923111330896e33f044212e7215102e9d9"
Jul 28 22:21:00.723087 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.723045    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:21:00.859462 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.859430    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8cdd\" (UniqueName: \"kubernetes.io/projected/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-kube-api-access-h8cdd\") pod \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\" (UID: \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\") "
Jul 28 22:21:00.859638 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.859469    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-tls-cert\") pod \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\" (UID: \"dbecf44c-45e0-4e3e-9619-eb9a0afabfd4\") "
Jul 28 22:21:00.861464 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.861429    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-kube-api-access-h8cdd" (OuterVolumeSpecName: "kube-api-access-h8cdd") pod "dbecf44c-45e0-4e3e-9619-eb9a0afabfd4" (UID: "dbecf44c-45e0-4e3e-9619-eb9a0afabfd4"). InnerVolumeSpecName "kube-api-access-h8cdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:21:00.869641 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.869612    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dbecf44c-45e0-4e3e-9619-eb9a0afabfd4" (UID: "dbecf44c-45e0-4e3e-9619-eb9a0afabfd4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:21:00.961064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.961022    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8cdd\" (UniqueName: \"kubernetes.io/projected/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-kube-api-access-h8cdd\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:21:00.961064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:00.961061    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4-tls-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:21:01.725833 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:01.725802    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-549599695d-kdq7j"
Jul 28 22:21:01.749976 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:01.749942    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-549599695d-kdq7j"]
Jul 28 22:21:01.754985 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:01.754960    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-549599695d-kdq7j"]
Jul 28 22:21:02.407150 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:02.407111    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbecf44c-45e0-4e3e-9619-eb9a0afabfd4" path="/var/lib/kubelet/pods/dbecf44c-45e0-4e3e-9619-eb9a0afabfd4/volumes"
Jul 28 22:21:08.876414 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:08.876374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:21:20.894719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:20.894677    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:21:20.949377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:20.949344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:21:30.217574 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:30.217533    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:21:40.835902 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:21:40.835842    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:04.889210 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:04.889166    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:06.138715 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:06.138676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:15.863462 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:15.863432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:15.921528 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:15.921489    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:47.721300 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:47.721262    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:47.821327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:47.821299    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:47.930352 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:47.930318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:53.534935 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:53.534894    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:53.640432 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:53.640402    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:53.734787 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:53.734756    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:53.886901 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:53.886812    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:59.538476 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:59.538446    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:23:59.601431 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:23:59.601400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:24:06.730787 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:24:06.730752    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:24:14.375865 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:24:14.375829    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:24:36.059590 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:24:36.059551    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:24:46.953481 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:24:46.953441    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:24:58.355053 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:24:58.354981    2560 scope.go:117] "RemoveContainer" containerID="83340c0afabec052e36bf0b6c62f9ead47123e6f868db6da7bfb82477ac0958d"
Jul 28 22:29:54.146136 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:29:54.146085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:30:00.134745 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.134708    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p"]
Jul 28 22:30:00.135159 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.135111    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbecf44c-45e0-4e3e-9619-eb9a0afabfd4" containerName="authorino"
Jul 28 22:30:00.135159 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.135123    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbecf44c-45e0-4e3e-9619-eb9a0afabfd4" containerName="authorino"
Jul 28 22:30:00.135243 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.135205    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbecf44c-45e0-4e3e-9619-eb9a0afabfd4" containerName="authorino"
Jul 28 22:30:00.138269 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.138252    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p"
Jul 28 22:30:00.140796 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.140769    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 22:30:00.145087 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.145062    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p"]
Jul 28 22:30:00.150307 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.150286    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p"
Jul 28 22:30:00.491903 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.491880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p"]
Jul 28 22:30:00.494147 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:30:00.494118    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4a20b5_7d2f_4c75_ac6c_b9a41db04bfc.slice/crio-ab3ba5778c0cbbf396718c4119fc4b461216b173965e0829c8cbf99c502b267b WatchSource:0}: Error finding container ab3ba5778c0cbbf396718c4119fc4b461216b173965e0829c8cbf99c502b267b: Status 404 returned error can't find the container with id ab3ba5778c0cbbf396718c4119fc4b461216b173965e0829c8cbf99c502b267b
Jul 28 22:30:00.496037 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:00.496020    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:30:01.358119 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:01.358074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p" event={"ID":"3d4a20b5-7d2f-4c75-ac6c-b9a41db04bfc","Type":"ContainerStarted","Data":"ab3ba5778c0cbbf396718c4119fc4b461216b173965e0829c8cbf99c502b267b"}
Jul 28 22:30:04.375151 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:04.375117    2560 generic.go:356] "Generic (PLEG): container finished" podID="3d4a20b5-7d2f-4c75-ac6c-b9a41db04bfc" containerID="04a09e998f5a5a759bae9c3dd13b35077f0f1b271ac83f2594ee87bb0bdc6f92" exitCode=0
Jul 28 22:30:04.375542 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:04.375190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p" event={"ID":"3d4a20b5-7d2f-4c75-ac6c-b9a41db04bfc","Type":"ContainerDied","Data":"04a09e998f5a5a759bae9c3dd13b35077f0f1b271ac83f2594ee87bb0bdc6f92"}
Jul 28 22:30:05.510373 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:05.510347    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p"
Jul 28 22:30:06.386277 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:06.386242    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p" event={"ID":"3d4a20b5-7d2f-4c75-ac6c-b9a41db04bfc","Type":"ContainerDied","Data":"ab3ba5778c0cbbf396718c4119fc4b461216b173965e0829c8cbf99c502b267b"}
Jul 28 22:30:06.386452 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:06.386290    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab3ba5778c0cbbf396718c4119fc4b461216b173965e0829c8cbf99c502b267b"
Jul 28 22:30:06.386452 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:06.386249    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754630-qg49p"
Jul 28 22:30:27.183721 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:30:27.183686    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:31:55.279030 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.278997    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"]
Jul 28 22:31:55.279513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.279426    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d4a20b5-7d2f-4c75-ac6c-b9a41db04bfc" containerName="cleanup"
Jul 28 22:31:55.279513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.279438    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d4a20b5-7d2f-4c75-ac6c-b9a41db04bfc" containerName="cleanup"
Jul 28 22:31:55.279586 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.279519    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d4a20b5-7d2f-4c75-ac6c-b9a41db04bfc" containerName="cleanup"
Jul 28 22:31:55.282594 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.282579    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.285052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.285033    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c88e6463-kserve-self-signed-certs\""
Jul 28 22:31:55.301571 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.301546    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"]
Jul 28 22:31:55.303601 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.303577    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:31:55.396512 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.396477    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d09633-6b05-4517-a7a7-61f9e438df58-tls-certs\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.396512 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.396515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-dshm\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.396719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.396577    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-model-cache\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.396719 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.396662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-home\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.396791 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.396737    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-tmp-dir\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.396791 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.396759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvhxh\" (UniqueName: \"kubernetes.io/projected/e1d09633-6b05-4517-a7a7-61f9e438df58-kube-api-access-fvhxh\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.498085 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.498048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-tmp-dir\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.498085 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.498085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvhxh\" (UniqueName: \"kubernetes.io/projected/e1d09633-6b05-4517-a7a7-61f9e438df58-kube-api-access-fvhxh\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.498332 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.498149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d09633-6b05-4517-a7a7-61f9e438df58-tls-certs\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.498332 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.498171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-dshm\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.498332 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.498207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-model-cache\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.498332 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.498257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-home\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.498535 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.498503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-tmp-dir\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.498628 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.498608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-home\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.498755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.498735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-model-cache\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.500416 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.500395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-dshm\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.500716 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.500683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d09633-6b05-4517-a7a7-61f9e438df58-tls-certs\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.505624 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.505606    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvhxh\" (UniqueName: \"kubernetes.io/projected/e1d09633-6b05-4517-a7a7-61f9e438df58-kube-api-access-fvhxh\") pod \"e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") " pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.592981 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.592898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:31:55.726896 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.726871    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"]
Jul 28 22:31:55.728910 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:31:55.728884    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1d09633_6b05_4517_a7a7_61f9e438df58.slice/crio-96cfb90ebc64d75135c5c9efce1694d74545071616dad57e5ca46883cc986f64 WatchSource:0}: Error finding container 96cfb90ebc64d75135c5c9efce1694d74545071616dad57e5ca46883cc986f64: Status 404 returned error can't find the container with id 96cfb90ebc64d75135c5c9efce1694d74545071616dad57e5ca46883cc986f64
Jul 28 22:31:55.940077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.939984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l" event={"ID":"e1d09633-6b05-4517-a7a7-61f9e438df58","Type":"ContainerStarted","Data":"6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55"}
Jul 28 22:31:55.940077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.940032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l" event={"ID":"e1d09633-6b05-4517-a7a7-61f9e438df58","Type":"ContainerStarted","Data":"96cfb90ebc64d75135c5c9efce1694d74545071616dad57e5ca46883cc986f64"}
Jul 28 22:31:55.958041 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:31:55.957996    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l" podStartSLOduration=0.95797987 podStartE2EDuration="957.97987ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:31:55.956375239 +0000 UTC m=+1318.078723158" watchObservedRunningTime="2026-07-28 22:31:55.95797987 +0000 UTC m=+1318.080327763"
Jul 28 22:32:05.593325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:05.593237    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:32:05.593325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:05.593292    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:32:05.605708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:05.605678    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:32:06.001369 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:06.001334    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:32:09.576270 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.576225    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"]
Jul 28 22:32:09.585046 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.585020    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.588970 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.588941    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c88e6463-kserve-self-signed-certs\""
Jul 28 22:32:09.590082 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.590057    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"]
Jul 28 22:32:09.597254 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.597232    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:32:09.634478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.634440    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gzg\" (UniqueName: \"kubernetes.io/projected/754b0c05-9bf6-4875-add1-8ff3d25990e6-kube-api-access-24gzg\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.634652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.634498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-model-cache\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.634652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.634571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754b0c05-9bf6-4875-add1-8ff3d25990e6-tls-certs\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.634652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.634593    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-tmp-dir\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.634652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.634618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-dshm\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.634652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.634642    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-home\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.735776 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.735733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gzg\" (UniqueName: \"kubernetes.io/projected/754b0c05-9bf6-4875-add1-8ff3d25990e6-kube-api-access-24gzg\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.735965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.735797    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-model-cache\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.735965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.735828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754b0c05-9bf6-4875-add1-8ff3d25990e6-tls-certs\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.735965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.735848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-tmp-dir\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.735965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.735866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-dshm\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.735965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.735886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-home\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.736369 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.736344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-model-cache\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.736458 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.736354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-home\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.736458 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.736434    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-tmp-dir\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.738225 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.738200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-dshm\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.738393 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.738375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754b0c05-9bf6-4875-add1-8ff3d25990e6-tls-certs\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.743269 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.743248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gzg\" (UniqueName: \"kubernetes.io/projected/754b0c05-9bf6-4875-add1-8ff3d25990e6-kube-api-access-24gzg\") pod \"e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") " pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:09.895260 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:09.895161    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:10.034156 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:10.034129    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"]
Jul 28 22:32:10.036002 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:32:10.035971    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod754b0c05_9bf6_4875_add1_8ff3d25990e6.slice/crio-07c0a35bad3285f43a093b28af00cd4216d45123775f88d87e9a4d9c31871b48 WatchSource:0}: Error finding container 07c0a35bad3285f43a093b28af00cd4216d45123775f88d87e9a4d9c31871b48: Status 404 returned error can't find the container with id 07c0a35bad3285f43a093b28af00cd4216d45123775f88d87e9a4d9c31871b48
Jul 28 22:32:11.017950 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:11.017909    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg" event={"ID":"754b0c05-9bf6-4875-add1-8ff3d25990e6","Type":"ContainerStarted","Data":"1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01"}
Jul 28 22:32:11.018445 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:11.017959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg" event={"ID":"754b0c05-9bf6-4875-add1-8ff3d25990e6","Type":"ContainerStarted","Data":"07c0a35bad3285f43a093b28af00cd4216d45123775f88d87e9a4d9c31871b48"}
Jul 28 22:32:11.036923 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:11.036878    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg" podStartSLOduration=2.036862684 podStartE2EDuration="2.036862684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:32:11.034636276 +0000 UTC m=+1333.156984171" watchObservedRunningTime="2026-07-28 22:32:11.036862684 +0000 UTC m=+1333.159210576"
Jul 28 22:32:19.895602 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:19.895568    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:19.896016 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:19.895894    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:19.907667 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:19.907639    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:20.076690 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:20.076622    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:31.279786 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:31.279751    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"]
Jul 28 22:32:31.299375 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:31.299345    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:32:32.692068 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:32.692039    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"]
Jul 28 22:32:32.713497 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:32.713467    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:32:46.298936 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.298885    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l" podUID="e1d09633-6b05-4517-a7a7-61f9e438df58" containerName="main" containerID="cri-o://6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55" gracePeriod=45
Jul 28 22:32:46.548266 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.548239    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:32:46.597809 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.597743    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-home\") pod \"e1d09633-6b05-4517-a7a7-61f9e438df58\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") "
Jul 28 22:32:46.597966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.597814    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-model-cache\") pod \"e1d09633-6b05-4517-a7a7-61f9e438df58\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") "
Jul 28 22:32:46.597966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.597870    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-dshm\") pod \"e1d09633-6b05-4517-a7a7-61f9e438df58\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") "
Jul 28 22:32:46.597966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.597894    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-tmp-dir\") pod \"e1d09633-6b05-4517-a7a7-61f9e438df58\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") "
Jul 28 22:32:46.597966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.597924    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvhxh\" (UniqueName: \"kubernetes.io/projected/e1d09633-6b05-4517-a7a7-61f9e438df58-kube-api-access-fvhxh\") pod \"e1d09633-6b05-4517-a7a7-61f9e438df58\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") "
Jul 28 22:32:46.597966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.597958    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d09633-6b05-4517-a7a7-61f9e438df58-tls-certs\") pod \"e1d09633-6b05-4517-a7a7-61f9e438df58\" (UID: \"e1d09633-6b05-4517-a7a7-61f9e438df58\") "
Jul 28 22:32:46.598229 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.598067    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-home" (OuterVolumeSpecName: "home") pod "e1d09633-6b05-4517-a7a7-61f9e438df58" (UID: "e1d09633-6b05-4517-a7a7-61f9e438df58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:32:46.598229 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.598090    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1d09633-6b05-4517-a7a7-61f9e438df58" (UID: "e1d09633-6b05-4517-a7a7-61f9e438df58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:32:46.598229 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.598190    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1d09633-6b05-4517-a7a7-61f9e438df58" (UID: "e1d09633-6b05-4517-a7a7-61f9e438df58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:32:46.598344 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.598335    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:46.598379 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.598352    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:46.598379 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.598366    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:46.600225 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.600193    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d09633-6b05-4517-a7a7-61f9e438df58-kube-api-access-fvhxh" (OuterVolumeSpecName: "kube-api-access-fvhxh") pod "e1d09633-6b05-4517-a7a7-61f9e438df58" (UID: "e1d09633-6b05-4517-a7a7-61f9e438df58"). InnerVolumeSpecName "kube-api-access-fvhxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:32:46.600340 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.600234    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-dshm" (OuterVolumeSpecName: "dshm") pod "e1d09633-6b05-4517-a7a7-61f9e438df58" (UID: "e1d09633-6b05-4517-a7a7-61f9e438df58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:32:46.600340 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.600229    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1d09633-6b05-4517-a7a7-61f9e438df58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1d09633-6b05-4517-a7a7-61f9e438df58" (UID: "e1d09633-6b05-4517-a7a7-61f9e438df58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:32:46.699085 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.699058    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d09633-6b05-4517-a7a7-61f9e438df58-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:46.699085 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.699083    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvhxh\" (UniqueName: \"kubernetes.io/projected/e1d09633-6b05-4517-a7a7-61f9e438df58-kube-api-access-fvhxh\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:46.699297 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:46.699123    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d09633-6b05-4517-a7a7-61f9e438df58-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:47.197579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.197543    2560 generic.go:356] "Generic (PLEG): container finished" podID="e1d09633-6b05-4517-a7a7-61f9e438df58" containerID="6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55" exitCode=0
Jul 28 22:32:47.197741 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.197587    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"
Jul 28 22:32:47.197741 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.197607    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l" event={"ID":"e1d09633-6b05-4517-a7a7-61f9e438df58","Type":"ContainerDied","Data":"6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55"}
Jul 28 22:32:47.197741 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.197646    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l" event={"ID":"e1d09633-6b05-4517-a7a7-61f9e438df58","Type":"ContainerDied","Data":"96cfb90ebc64d75135c5c9efce1694d74545071616dad57e5ca46883cc986f64"}
Jul 28 22:32:47.197741 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.197667    2560 scope.go:117] "RemoveContainer" containerID="6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55"
Jul 28 22:32:47.207483 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.207461    2560 scope.go:117] "RemoveContainer" containerID="6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55"
Jul 28 22:32:47.207745 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:32:47.207724    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55\": container with ID starting with 6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55 not found: ID does not exist" containerID="6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55"
Jul 28 22:32:47.207796 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.207755    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55"} err="failed to get container status \"6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55\": rpc error: code = NotFound desc = could not find container \"6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55\": container with ID starting with 6f1a8c4e78b32cf40979c7ce760faecf9b2fbdc5401b9ef8805555468c4cbc55 not found: ID does not exist"
Jul 28 22:32:47.219295 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.219274    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"]
Jul 28 22:32:47.222650 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.222618    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c88e6463-kserve-594ccdfbcf-ssh7l"]
Jul 28 22:32:47.711227 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.711188    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg" podUID="754b0c05-9bf6-4875-add1-8ff3d25990e6" containerName="main" containerID="cri-o://1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01" gracePeriod=45
Jul 28 22:32:47.846341 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.846311    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:32:47.968172 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:47.968148    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:48.011378 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011332    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-tmp-dir\") pod \"754b0c05-9bf6-4875-add1-8ff3d25990e6\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") "
Jul 28 22:32:48.011549 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011392    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-home\") pod \"754b0c05-9bf6-4875-add1-8ff3d25990e6\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") "
Jul 28 22:32:48.011549 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011472    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-dshm\") pod \"754b0c05-9bf6-4875-add1-8ff3d25990e6\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") "
Jul 28 22:32:48.011549 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011496    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-model-cache\") pod \"754b0c05-9bf6-4875-add1-8ff3d25990e6\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") "
Jul 28 22:32:48.011549 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011537    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24gzg\" (UniqueName: \"kubernetes.io/projected/754b0c05-9bf6-4875-add1-8ff3d25990e6-kube-api-access-24gzg\") pod \"754b0c05-9bf6-4875-add1-8ff3d25990e6\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") "
Jul 28 22:32:48.011837 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011577    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754b0c05-9bf6-4875-add1-8ff3d25990e6-tls-certs\") pod \"754b0c05-9bf6-4875-add1-8ff3d25990e6\" (UID: \"754b0c05-9bf6-4875-add1-8ff3d25990e6\") "
Jul 28 22:32:48.011837 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011632    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "754b0c05-9bf6-4875-add1-8ff3d25990e6" (UID: "754b0c05-9bf6-4875-add1-8ff3d25990e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:32:48.011837 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011644    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-home" (OuterVolumeSpecName: "home") pod "754b0c05-9bf6-4875-add1-8ff3d25990e6" (UID: "754b0c05-9bf6-4875-add1-8ff3d25990e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:32:48.011837 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011778    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "754b0c05-9bf6-4875-add1-8ff3d25990e6" (UID: "754b0c05-9bf6-4875-add1-8ff3d25990e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:32:48.012030 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011972    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:48.012030 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.011989    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:48.012030 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.012003    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:48.013709 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.013684    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/754b0c05-9bf6-4875-add1-8ff3d25990e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "754b0c05-9bf6-4875-add1-8ff3d25990e6" (UID: "754b0c05-9bf6-4875-add1-8ff3d25990e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:32:48.014061 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.014038    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-dshm" (OuterVolumeSpecName: "dshm") pod "754b0c05-9bf6-4875-add1-8ff3d25990e6" (UID: "754b0c05-9bf6-4875-add1-8ff3d25990e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:32:48.014174 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.014088    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754b0c05-9bf6-4875-add1-8ff3d25990e6-kube-api-access-24gzg" (OuterVolumeSpecName: "kube-api-access-24gzg") pod "754b0c05-9bf6-4875-add1-8ff3d25990e6" (UID: "754b0c05-9bf6-4875-add1-8ff3d25990e6"). InnerVolumeSpecName "kube-api-access-24gzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:32:48.113150 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.113086    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754b0c05-9bf6-4875-add1-8ff3d25990e6-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:48.113150 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.113146    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754b0c05-9bf6-4875-add1-8ff3d25990e6-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:48.113150 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.113156    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24gzg\" (UniqueName: \"kubernetes.io/projected/754b0c05-9bf6-4875-add1-8ff3d25990e6-kube-api-access-24gzg\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:32:48.204384 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.204349    2560 generic.go:356] "Generic (PLEG): container finished" podID="754b0c05-9bf6-4875-add1-8ff3d25990e6" containerID="1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01" exitCode=0
Jul 28 22:32:48.204533 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.204391    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"
Jul 28 22:32:48.204533 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.204435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg" event={"ID":"754b0c05-9bf6-4875-add1-8ff3d25990e6","Type":"ContainerDied","Data":"1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01"}
Jul 28 22:32:48.204533 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.204474    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg" event={"ID":"754b0c05-9bf6-4875-add1-8ff3d25990e6","Type":"ContainerDied","Data":"07c0a35bad3285f43a093b28af00cd4216d45123775f88d87e9a4d9c31871b48"}
Jul 28 22:32:48.204533 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.204491    2560 scope.go:117] "RemoveContainer" containerID="1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01"
Jul 28 22:32:48.213917 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.213898    2560 scope.go:117] "RemoveContainer" containerID="1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01"
Jul 28 22:32:48.214255 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:32:48.214235    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01\": container with ID starting with 1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01 not found: ID does not exist" containerID="1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01"
Jul 28 22:32:48.214329 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.214264    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01"} err="failed to get container status \"1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01\": rpc error: code = NotFound desc = could not find container \"1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01\": container with ID starting with 1189e8190cdb6a2136ae5c4b9751319c441062a85ee26bc3ff48f6ce625deb01 not found: ID does not exist"
Jul 28 22:32:48.225120 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.225039    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"]
Jul 28 22:32:48.229425 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.229401    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c88e6463-kserve-7fc6c7687-sdndg"]
Jul 28 22:32:48.407950 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.407915    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="754b0c05-9bf6-4875-add1-8ff3d25990e6" path="/var/lib/kubelet/pods/754b0c05-9bf6-4875-add1-8ff3d25990e6/volumes"
Jul 28 22:32:48.408356 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:32:48.408341    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1d09633-6b05-4517-a7a7-61f9e438df58" path="/var/lib/kubelet/pods/e1d09633-6b05-4517-a7a7-61f9e438df58/volumes"
Jul 28 22:33:17.773826 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:33:17.773785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:33:17.873712 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:33:17.873683    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:35:41.005465 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:35:41.005383    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:01.893733 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:01.893691    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:04.970083 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:04.970040    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:05.072505 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:05.072470    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:05.420773 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:05.420668    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:11.094036 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:11.093998    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:11.193513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:11.193479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:11.316432 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:11.316393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:11.626423 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:11.626390    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:11.811642 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:11.811610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:13.875671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:13.875636    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:13.949909 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:13.949845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:14.186805 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.186712    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:14.379815 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.379778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:14.713404 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.713369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:14.813398 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.813360    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:14.821130 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.821078    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"]
Jul 28 22:36:14.821486 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.821474    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754b0c05-9bf6-4875-add1-8ff3d25990e6" containerName="main"
Jul 28 22:36:14.821530 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.821487    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="754b0c05-9bf6-4875-add1-8ff3d25990e6" containerName="main"
Jul 28 22:36:14.821530 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.821498    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1d09633-6b05-4517-a7a7-61f9e438df58" containerName="main"
Jul 28 22:36:14.821530 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.821504    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1d09633-6b05-4517-a7a7-61f9e438df58" containerName="main"
Jul 28 22:36:14.821628 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.821581    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="754b0c05-9bf6-4875-add1-8ff3d25990e6" containerName="main"
Jul 28 22:36:14.821628 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.821591    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1d09633-6b05-4517-a7a7-61f9e438df58" containerName="main"
Jul 28 22:36:14.824507 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.824489    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:14.827450 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.827429    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c67c1cbd-dockercfg-gtw2h\""
Jul 28 22:36:14.827564 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.827432    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c67c1cbd\""
Jul 28 22:36:14.835821 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.835798    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"]
Jul 28 22:36:14.857836 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.857796    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"]
Jul 28 22:36:14.864254 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.863338    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:14.872277 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.872246    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"]
Jul 28 22:36:14.993346 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.993311    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7eba8-b73f-4017-b617-15aebcb0a733-plugins-config-volume\") pod \"payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2\" (UID: \"94e7eba8-b73f-4017-b617-15aebcb0a733\") " pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:14.993346 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.993350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rhd\" (UniqueName: \"kubernetes.io/projected/1190f146-6efd-4ced-bb25-84e80cb95398-kube-api-access-29rhd\") pod \"payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4\" (UID: \"1190f146-6efd-4ced-bb25-84e80cb95398\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:14.993949 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.993423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxr4w\" (UniqueName: \"kubernetes.io/projected/94e7eba8-b73f-4017-b617-15aebcb0a733-kube-api-access-fxr4w\") pod \"payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2\" (UID: \"94e7eba8-b73f-4017-b617-15aebcb0a733\") " pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:14.993949 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:14.993473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190f146-6efd-4ced-bb25-84e80cb95398-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4\" (UID: \"1190f146-6efd-4ced-bb25-84e80cb95398\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:15.094155 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.094082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7eba8-b73f-4017-b617-15aebcb0a733-plugins-config-volume\") pod \"payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2\" (UID: \"94e7eba8-b73f-4017-b617-15aebcb0a733\") " pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:15.094347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.094167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rhd\" (UniqueName: \"kubernetes.io/projected/1190f146-6efd-4ced-bb25-84e80cb95398-kube-api-access-29rhd\") pod \"payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4\" (UID: \"1190f146-6efd-4ced-bb25-84e80cb95398\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:15.094347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.094218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxr4w\" (UniqueName: \"kubernetes.io/projected/94e7eba8-b73f-4017-b617-15aebcb0a733-kube-api-access-fxr4w\") pod \"payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2\" (UID: \"94e7eba8-b73f-4017-b617-15aebcb0a733\") " pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:15.094347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.094244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190f146-6efd-4ced-bb25-84e80cb95398-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4\" (UID: \"1190f146-6efd-4ced-bb25-84e80cb95398\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:15.094820 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.094798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7eba8-b73f-4017-b617-15aebcb0a733-plugins-config-volume\") pod \"payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2\" (UID: \"94e7eba8-b73f-4017-b617-15aebcb0a733\") " pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:15.094963 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.094946    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190f146-6efd-4ced-bb25-84e80cb95398-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4\" (UID: \"1190f146-6efd-4ced-bb25-84e80cb95398\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:15.104784 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.104756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxr4w\" (UniqueName: \"kubernetes.io/projected/94e7eba8-b73f-4017-b617-15aebcb0a733-kube-api-access-fxr4w\") pod \"payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2\" (UID: \"94e7eba8-b73f-4017-b617-15aebcb0a733\") " pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:15.104895 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.104762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rhd\" (UniqueName: \"kubernetes.io/projected/1190f146-6efd-4ced-bb25-84e80cb95398-kube-api-access-29rhd\") pod \"payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4\" (UID: \"1190f146-6efd-4ced-bb25-84e80cb95398\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:15.136457 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.136418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:15.175358 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.175324    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:15.288253 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.288221    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"]
Jul 28 22:36:15.291694 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:36:15.291665    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1190f146_6efd_4ced_bb25_84e80cb95398.slice/crio-a474d26971155e55dd70f01ac9533241c1f3e9b02fc005fc58948cb180393c03 WatchSource:0}: Error finding container a474d26971155e55dd70f01ac9533241c1f3e9b02fc005fc58948cb180393c03: Status 404 returned error can't find the container with id a474d26971155e55dd70f01ac9533241c1f3e9b02fc005fc58948cb180393c03
Jul 28 22:36:15.336426 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:15.336399    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"]
Jul 28 22:36:15.338856 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:36:15.338829    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e7eba8_b73f_4017_b617_15aebcb0a733.slice/crio-be6a230147b88d007fb240ab37aedcb3867965e4cd19f0c94348bb4f11d535c3 WatchSource:0}: Error finding container be6a230147b88d007fb240ab37aedcb3867965e4cd19f0c94348bb4f11d535c3: Status 404 returned error can't find the container with id be6a230147b88d007fb240ab37aedcb3867965e4cd19f0c94348bb4f11d535c3
Jul 28 22:36:16.243299 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:16.243261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2" event={"ID":"94e7eba8-b73f-4017-b617-15aebcb0a733","Type":"ContainerStarted","Data":"9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d"}
Jul 28 22:36:16.243781 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:16.243315    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2" event={"ID":"94e7eba8-b73f-4017-b617-15aebcb0a733","Type":"ContainerStarted","Data":"be6a230147b88d007fb240ab37aedcb3867965e4cd19f0c94348bb4f11d535c3"}
Jul 28 22:36:16.243781 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:16.243385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:16.244913 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:16.244891    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4" event={"ID":"1190f146-6efd-4ced-bb25-84e80cb95398","Type":"ContainerStarted","Data":"a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d"}
Jul 28 22:36:16.245007 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:16.244929    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4" event={"ID":"1190f146-6efd-4ced-bb25-84e80cb95398","Type":"ContainerStarted","Data":"a474d26971155e55dd70f01ac9533241c1f3e9b02fc005fc58948cb180393c03"}
Jul 28 22:36:16.245007 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:16.244962    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:16.261557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:16.261503    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2" podStartSLOduration=2.2614872569999998 podStartE2EDuration="2.261487257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:16.259701459 +0000 UTC m=+1578.382049352" watchObservedRunningTime="2026-07-28 22:36:16.261487257 +0000 UTC m=+1578.383835149"
Jul 28 22:36:16.275405 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:16.275363    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4" podStartSLOduration=2.2753462 podStartE2EDuration="2.2753462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:16.274120581 +0000 UTC m=+1578.396468495" watchObservedRunningTime="2026-07-28 22:36:16.2753462 +0000 UTC m=+1578.397694092"
Jul 28 22:36:19.518288 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:19.518249    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:27.253295 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:27.253262    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:27.253715 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:27.253702    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:39.379006 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:39.378976    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"]
Jul 28 22:36:39.382584 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:39.382569    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"
Jul 28 22:36:39.385158 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:39.385135    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 22:36:39.389686 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:39.389661    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"]
Jul 28 22:36:39.432076 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:39.432032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a6f55-b00c-4d70-ae4c-6bdf3d5a821d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99\" (UID: \"237a6f55-b00c-4d70-ae4c-6bdf3d5a821d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"
Jul 28 22:36:39.532814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:39.532768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a6f55-b00c-4d70-ae4c-6bdf3d5a821d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99\" (UID: \"237a6f55-b00c-4d70-ae4c-6bdf3d5a821d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"
Jul 28 22:36:39.533501 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:39.533479    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a6f55-b00c-4d70-ae4c-6bdf3d5a821d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99\" (UID: \"237a6f55-b00c-4d70-ae4c-6bdf3d5a821d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"
Jul 28 22:36:39.696050 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:39.695954    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"
Jul 28 22:36:40.035972 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:40.035886    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"]
Jul 28 22:36:40.038502 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:36:40.038475    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237a6f55_b00c_4d70_ae4c_6bdf3d5a821d.slice/crio-ece1ba88a60cc17cb14d92ab55f99fbef4c1278f08743389643e7635b2f7f16f WatchSource:0}: Error finding container ece1ba88a60cc17cb14d92ab55f99fbef4c1278f08743389643e7635b2f7f16f: Status 404 returned error can't find the container with id ece1ba88a60cc17cb14d92ab55f99fbef4c1278f08743389643e7635b2f7f16f
Jul 28 22:36:40.373532 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:40.373493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99" event={"ID":"237a6f55-b00c-4d70-ae4c-6bdf3d5a821d","Type":"ContainerStarted","Data":"d211253ec770a7b16561101cd51215692ca8ae85914de90a2aa0b3208d270c5f"}
Jul 28 22:36:40.373707 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:40.373553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99" event={"ID":"237a6f55-b00c-4d70-ae4c-6bdf3d5a821d","Type":"ContainerStarted","Data":"ece1ba88a60cc17cb14d92ab55f99fbef4c1278f08743389643e7635b2f7f16f"}
Jul 28 22:36:40.389305 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:40.389254    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99" podStartSLOduration=1.3892373359999999 podStartE2EDuration="1.389237336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:40.387430039 +0000 UTC m=+1602.509777961" watchObservedRunningTime="2026-07-28 22:36:40.389237336 +0000 UTC m=+1602.511585228"
Jul 28 22:36:41.380368 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:41.380335    2560 generic.go:356] "Generic (PLEG): container finished" podID="237a6f55-b00c-4d70-ae4c-6bdf3d5a821d" containerID="d211253ec770a7b16561101cd51215692ca8ae85914de90a2aa0b3208d270c5f" exitCode=0
Jul 28 22:36:41.380579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:41.380380    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99" event={"ID":"237a6f55-b00c-4d70-ae4c-6bdf3d5a821d","Type":"ContainerDied","Data":"d211253ec770a7b16561101cd51215692ca8ae85914de90a2aa0b3208d270c5f"}
Jul 28 22:36:42.520766 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:42.520739    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"
Jul 28 22:36:42.663122 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:42.663006    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a6f55-b00c-4d70-ae4c-6bdf3d5a821d-maas-api-service-ca\") pod \"237a6f55-b00c-4d70-ae4c-6bdf3d5a821d\" (UID: \"237a6f55-b00c-4d70-ae4c-6bdf3d5a821d\") "
Jul 28 22:36:42.663392 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:42.663358    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237a6f55-b00c-4d70-ae4c-6bdf3d5a821d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "237a6f55-b00c-4d70-ae4c-6bdf3d5a821d" (UID: "237a6f55-b00c-4d70-ae4c-6bdf3d5a821d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:36:42.764291 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:42.764256    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a6f55-b00c-4d70-ae4c-6bdf3d5a821d-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:36:43.393702 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:43.393669    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"
Jul 28 22:36:43.393702 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:43.393689    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99" event={"ID":"237a6f55-b00c-4d70-ae4c-6bdf3d5a821d","Type":"ContainerDied","Data":"ece1ba88a60cc17cb14d92ab55f99fbef4c1278f08743389643e7635b2f7f16f"}
Jul 28 22:36:43.393956 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:43.393757    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ece1ba88a60cc17cb14d92ab55f99fbef4c1278f08743389643e7635b2f7f16f"
Jul 28 22:36:44.719439 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.719401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:44.746455 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.746419    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"]
Jul 28 22:36:44.746702 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.746677    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2" podUID="94e7eba8-b73f-4017-b617-15aebcb0a733" containerName="payload-processing" containerID="cri-o://9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d" gracePeriod=30
Jul 28 22:36:44.751154 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.750787    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"]
Jul 28 22:36:44.751154 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.750997    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4" podUID="1190f146-6efd-4ced-bb25-84e80cb95398" containerName="payload-pre-processing" containerID="cri-o://a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d" gracePeriod=30
Jul 28 22:36:44.787331 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.787293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:44.889410 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.889367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:44.891206 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.891177    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"]
Jul 28 22:36:44.893057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.893033    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c67c1cbd-jkd99"]
Jul 28 22:36:44.987873 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:44.987783    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:45.038297 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.038271    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:45.041330 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.041307    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:45.050005 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.049979    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:45.187202 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.187173    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7eba8-b73f-4017-b617-15aebcb0a733-plugins-config-volume\") pod \"94e7eba8-b73f-4017-b617-15aebcb0a733\" (UID: \"94e7eba8-b73f-4017-b617-15aebcb0a733\") "
Jul 28 22:36:45.187393 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.187259    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190f146-6efd-4ced-bb25-84e80cb95398-plugins-config-volume\") pod \"1190f146-6efd-4ced-bb25-84e80cb95398\" (UID: \"1190f146-6efd-4ced-bb25-84e80cb95398\") "
Jul 28 22:36:45.187393 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.187317    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29rhd\" (UniqueName: \"kubernetes.io/projected/1190f146-6efd-4ced-bb25-84e80cb95398-kube-api-access-29rhd\") pod \"1190f146-6efd-4ced-bb25-84e80cb95398\" (UID: \"1190f146-6efd-4ced-bb25-84e80cb95398\") "
Jul 28 22:36:45.187393 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.187348    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxr4w\" (UniqueName: \"kubernetes.io/projected/94e7eba8-b73f-4017-b617-15aebcb0a733-kube-api-access-fxr4w\") pod \"94e7eba8-b73f-4017-b617-15aebcb0a733\" (UID: \"94e7eba8-b73f-4017-b617-15aebcb0a733\") "
Jul 28 22:36:45.187572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.187547    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e7eba8-b73f-4017-b617-15aebcb0a733-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94e7eba8-b73f-4017-b617-15aebcb0a733" (UID: "94e7eba8-b73f-4017-b617-15aebcb0a733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:36:45.187690 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.187666    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1190f146-6efd-4ced-bb25-84e80cb95398-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1190f146-6efd-4ced-bb25-84e80cb95398" (UID: "1190f146-6efd-4ced-bb25-84e80cb95398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:36:45.189622 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.189595    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e7eba8-b73f-4017-b617-15aebcb0a733-kube-api-access-fxr4w" (OuterVolumeSpecName: "kube-api-access-fxr4w") pod "94e7eba8-b73f-4017-b617-15aebcb0a733" (UID: "94e7eba8-b73f-4017-b617-15aebcb0a733"). InnerVolumeSpecName "kube-api-access-fxr4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:36:45.189746 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.189653    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1190f146-6efd-4ced-bb25-84e80cb95398-kube-api-access-29rhd" (OuterVolumeSpecName: "kube-api-access-29rhd") pod "1190f146-6efd-4ced-bb25-84e80cb95398" (UID: "1190f146-6efd-4ced-bb25-84e80cb95398"). InnerVolumeSpecName "kube-api-access-29rhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:36:45.273292 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.273203    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:45.288476 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.288447    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rhd\" (UniqueName: \"kubernetes.io/projected/1190f146-6efd-4ced-bb25-84e80cb95398-kube-api-access-29rhd\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:36:45.288476 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.288474    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxr4w\" (UniqueName: \"kubernetes.io/projected/94e7eba8-b73f-4017-b617-15aebcb0a733-kube-api-access-fxr4w\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:36:45.288650 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.288483    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7eba8-b73f-4017-b617-15aebcb0a733-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:36:45.288650 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.288493    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190f146-6efd-4ced-bb25-84e80cb95398-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:36:45.407641 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.407603    2560 generic.go:356] "Generic (PLEG): container finished" podID="94e7eba8-b73f-4017-b617-15aebcb0a733" containerID="9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d" exitCode=0
Jul 28 22:36:45.407827 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.407649    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"
Jul 28 22:36:45.407827 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.407699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2" event={"ID":"94e7eba8-b73f-4017-b617-15aebcb0a733","Type":"ContainerDied","Data":"9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d"}
Jul 28 22:36:45.407827 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.407745    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2" event={"ID":"94e7eba8-b73f-4017-b617-15aebcb0a733","Type":"ContainerDied","Data":"be6a230147b88d007fb240ab37aedcb3867965e4cd19f0c94348bb4f11d535c3"}
Jul 28 22:36:45.407827 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.407773    2560 scope.go:117] "RemoveContainer" containerID="9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d"
Jul 28 22:36:45.409051 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.409031    2560 generic.go:356] "Generic (PLEG): container finished" podID="1190f146-6efd-4ced-bb25-84e80cb95398" containerID="a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d" exitCode=0
Jul 28 22:36:45.409154 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.409072    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4" event={"ID":"1190f146-6efd-4ced-bb25-84e80cb95398","Type":"ContainerDied","Data":"a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d"}
Jul 28 22:36:45.409154 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.409142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4" event={"ID":"1190f146-6efd-4ced-bb25-84e80cb95398","Type":"ContainerDied","Data":"a474d26971155e55dd70f01ac9533241c1f3e9b02fc005fc58948cb180393c03"}
Jul 28 22:36:45.409302 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.409078    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"
Jul 28 22:36:45.418457 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.418438    2560 scope.go:117] "RemoveContainer" containerID="9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d"
Jul 28 22:36:45.418777 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:36:45.418756    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d\": container with ID starting with 9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d not found: ID does not exist" containerID="9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d"
Jul 28 22:36:45.418873 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.418784    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d"} err="failed to get container status \"9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d\": rpc error: code = NotFound desc = could not find container \"9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d\": container with ID starting with 9d5401cc7714a42bf1b11b398db2a5bc86da63311c44047bca9a10f6e151942d not found: ID does not exist"
Jul 28 22:36:45.418873 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.418801    2560 scope.go:117] "RemoveContainer" containerID="a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d"
Jul 28 22:36:45.427054 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.427024    2560 scope.go:117] "RemoveContainer" containerID="a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d"
Jul 28 22:36:45.427488 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:36:45.427432    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d\": container with ID starting with a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d not found: ID does not exist" containerID="a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d"
Jul 28 22:36:45.427488 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.427464    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d"} err="failed to get container status \"a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d\": rpc error: code = NotFound desc = could not find container \"a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d\": container with ID starting with a17acc01de2c584728714b5f4cc37bcd05a6066fc84e14e47e119fbfad15482d not found: ID does not exist"
Jul 28 22:36:45.429735 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.429711    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"]
Jul 28 22:36:45.433316 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.433294    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c67c1cbd-7b4ddc88cd-77pt2"]
Jul 28 22:36:45.443634 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.443608    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"]
Jul 28 22:36:45.447265 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.447242    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c67c1cbd-66fcf4774d-ltph4"]
Jul 28 22:36:45.528419 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.528323    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:45.627387 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.627353    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:45.735366 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:45.735333    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:46.052031 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:46.052001    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:46.244666 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:46.244627    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:46.407757 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:46.407672    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1190f146-6efd-4ced-bb25-84e80cb95398" path="/var/lib/kubelet/pods/1190f146-6efd-4ced-bb25-84e80cb95398/volumes"
Jul 28 22:36:46.408074 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:46.408049    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="237a6f55-b00c-4d70-ae4c-6bdf3d5a821d" path="/var/lib/kubelet/pods/237a6f55-b00c-4d70-ae4c-6bdf3d5a821d/volumes"
Jul 28 22:36:46.408498 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:46.408482    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e7eba8-b73f-4017-b617-15aebcb0a733" path="/var/lib/kubelet/pods/94e7eba8-b73f-4017-b617-15aebcb0a733/volumes"
Jul 28 22:36:48.105943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.105887    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:48.336392 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336358    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"]
Jul 28 22:36:48.336805 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336791    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="237a6f55-b00c-4d70-ae4c-6bdf3d5a821d" containerName="revoke-keys"
Jul 28 22:36:48.336864 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336808    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="237a6f55-b00c-4d70-ae4c-6bdf3d5a821d" containerName="revoke-keys"
Jul 28 22:36:48.336864 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336825    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1190f146-6efd-4ced-bb25-84e80cb95398" containerName="payload-pre-processing"
Jul 28 22:36:48.336864 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336830    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1190f146-6efd-4ced-bb25-84e80cb95398" containerName="payload-pre-processing"
Jul 28 22:36:48.336864 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336857    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e7eba8-b73f-4017-b617-15aebcb0a733" containerName="payload-processing"
Jul 28 22:36:48.336987 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336867    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e7eba8-b73f-4017-b617-15aebcb0a733" containerName="payload-processing"
Jul 28 22:36:48.336987 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336936    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e7eba8-b73f-4017-b617-15aebcb0a733" containerName="payload-processing"
Jul 28 22:36:48.336987 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336947    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1190f146-6efd-4ced-bb25-84e80cb95398" containerName="payload-pre-processing"
Jul 28 22:36:48.336987 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.336953    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="237a6f55-b00c-4d70-ae4c-6bdf3d5a821d" containerName="revoke-keys"
Jul 28 22:36:48.341354 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.341337    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:48.344943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.344918    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6ad9588a-serving-cert\""
Jul 28 22:36:48.345073 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.345011    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-89lsd\""
Jul 28 22:36:48.361407 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.361350    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"]
Jul 28 22:36:48.400112 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.400078    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:48.421878 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.421843    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930fa446-fef9-4637-965c-7a366958d102-openshift-service-ca\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:48.422046 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.421886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930fa446-fef9-4637-965c-7a366958d102-maas-api-tls\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:48.422088 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.422038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdr5m\" (UniqueName: \"kubernetes.io/projected/930fa446-fef9-4637-965c-7a366958d102-kube-api-access-tdr5m\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:48.522808 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.522765    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930fa446-fef9-4637-965c-7a366958d102-maas-api-tls\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:48.522979 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.522867    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdr5m\" (UniqueName: \"kubernetes.io/projected/930fa446-fef9-4637-965c-7a366958d102-kube-api-access-tdr5m\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:48.522979 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.522912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930fa446-fef9-4637-965c-7a366958d102-openshift-service-ca\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:48.522979 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:36:48.522933    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-serving-cert: secret "maas-api-e2e-migrate-6ad9588a-serving-cert" not found
Jul 28 22:36:48.523139 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:36:48.523030    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/930fa446-fef9-4637-965c-7a366958d102-maas-api-tls podName:930fa446-fef9-4637-965c-7a366958d102 nodeName:}" failed. No retries permitted until 2026-07-28 22:36:49.023006377 +0000 UTC m=+1611.145354248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/930fa446-fef9-4637-965c-7a366958d102-maas-api-tls") pod "maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x" (UID: "930fa446-fef9-4637-965c-7a366958d102") : secret "maas-api-e2e-migrate-6ad9588a-serving-cert" not found
Jul 28 22:36:48.523601 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.523582    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930fa446-fef9-4637-965c-7a366958d102-openshift-service-ca\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:48.531993 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.531970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdr5m\" (UniqueName: \"kubernetes.io/projected/930fa446-fef9-4637-965c-7a366958d102-kube-api-access-tdr5m\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:48.801247 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.801216    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:48.955571 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:48.955538    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:49.028041 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.028004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930fa446-fef9-4637-965c-7a366958d102-maas-api-tls\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:49.030530 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.030501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930fa446-fef9-4637-965c-7a366958d102-maas-api-tls\") pod \"maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:49.040901 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.040874    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"]
Jul 28 22:36:49.043898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.043874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:36:49.044902 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.044887    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:36:49.047469 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.047450    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6ad9588a\""
Jul 28 22:36:49.055943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.055920    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"]
Jul 28 22:36:49.128764 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.128725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh\" (UID: \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:36:49.129277 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.128851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k862p\" (UniqueName: \"kubernetes.io/projected/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-kube-api-access-k862p\") pod \"payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh\" (UID: \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:36:49.229420 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.229387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh\" (UID: \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:36:49.229588 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.229498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k862p\" (UniqueName: \"kubernetes.io/projected/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-kube-api-access-k862p\") pod \"payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh\" (UID: \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:36:49.230142 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.230093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh\" (UID: \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:36:49.237864 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.237840    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k862p\" (UniqueName: \"kubernetes.io/projected/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-kube-api-access-k862p\") pod \"payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh\" (UID: \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:36:49.251712 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.251687    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:49.357576 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.357544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:36:49.390645 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.390620    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"]
Jul 28 22:36:49.393334 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:36:49.393307    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930fa446_fef9_4637_965c_7a366958d102.slice/crio-583e6c3daeef7697126298e18d72f96330736171efea84735e41d47c4a4beed9 WatchSource:0}: Error finding container 583e6c3daeef7697126298e18d72f96330736171efea84735e41d47c4a4beed9: Status 404 returned error can't find the container with id 583e6c3daeef7697126298e18d72f96330736171efea84735e41d47c4a4beed9
Jul 28 22:36:49.394644 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.394625    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:36:49.441535 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.441504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x" event={"ID":"930fa446-fef9-4637-965c-7a366958d102","Type":"ContainerStarted","Data":"583e6c3daeef7697126298e18d72f96330736171efea84735e41d47c4a4beed9"}
Jul 28 22:36:49.502287 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:49.502257    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"]
Jul 28 22:36:49.503496 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:36:49.503469    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd8eff5_2ee7_487d_8726_aeca49a8bbbc.slice/crio-ec46b258b6d5b0a2292d8fb330eb10b3f5ba80f81def9178a47c9a9cd3e868f4 WatchSource:0}: Error finding container ec46b258b6d5b0a2292d8fb330eb10b3f5ba80f81def9178a47c9a9cd3e868f4: Status 404 returned error can't find the container with id ec46b258b6d5b0a2292d8fb330eb10b3f5ba80f81def9178a47c9a9cd3e868f4
Jul 28 22:36:50.449325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:50.449235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh" event={"ID":"edd8eff5-2ee7-487d-8726-aeca49a8bbbc","Type":"ContainerStarted","Data":"4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac"}
Jul 28 22:36:50.449325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:50.449293    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh" event={"ID":"edd8eff5-2ee7-487d-8726-aeca49a8bbbc","Type":"ContainerStarted","Data":"ec46b258b6d5b0a2292d8fb330eb10b3f5ba80f81def9178a47c9a9cd3e868f4"}
Jul 28 22:36:50.449786 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:50.449380    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:36:50.467983 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:50.467924    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh" podStartSLOduration=1.467903819 podStartE2EDuration="1.467903819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:50.465515118 +0000 UTC m=+1612.587863031" watchObservedRunningTime="2026-07-28 22:36:50.467903819 +0000 UTC m=+1612.590251714"
Jul 28 22:36:52.460739 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:52.460698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x" event={"ID":"930fa446-fef9-4637-965c-7a366958d102","Type":"ContainerStarted","Data":"52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d"}
Jul 28 22:36:52.461232 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:52.460772    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:36:52.478252 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:52.478209    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x" podStartSLOduration=2.256840666 podStartE2EDuration="4.478194859s" totalImagesPullingTime="2.221354193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:52.477189489 +0000 UTC m=+1614.599537397" watchObservedRunningTime="2026-07-28 22:36:52.478194859 +0000 UTC m=+1614.600542777"
Jul 28 22:36:58.471842 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:36:58.471812    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:37:01.456756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:01.456727    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:37:28.855593 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:28.855554    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"]
Jul 28 22:37:28.856043 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:28.855819    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x" podUID="930fa446-fef9-4637-965c-7a366958d102" containerName="maas-api" containerID="cri-o://52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d" gracePeriod=30
Jul 28 22:37:28.883224 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:28.883187    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:28.916666 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:28.916638    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"]
Jul 28 22:37:28.916885 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:28.916853    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh" podUID="edd8eff5-2ee7-487d-8726-aeca49a8bbbc" containerName="payload-pre-processing" containerID="cri-o://4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac" gracePeriod=30
Jul 28 22:37:28.982335 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:28.982305    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:29.083479 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.083423    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:29.138644 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.138612    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:29.160189 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.160162    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:37:29.199535 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.199508    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:37:29.216424 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.216387    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:29.311088 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.311059    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930fa446-fef9-4637-965c-7a366958d102-openshift-service-ca\") pod \"930fa446-fef9-4637-965c-7a366958d102\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") "
Jul 28 22:37:29.311301 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.311138    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-plugins-config-volume\") pod \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\" (UID: \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\") "
Jul 28 22:37:29.311301 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.311191    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k862p\" (UniqueName: \"kubernetes.io/projected/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-kube-api-access-k862p\") pod \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\" (UID: \"edd8eff5-2ee7-487d-8726-aeca49a8bbbc\") "
Jul 28 22:37:29.311301 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.311267    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdr5m\" (UniqueName: \"kubernetes.io/projected/930fa446-fef9-4637-965c-7a366958d102-kube-api-access-tdr5m\") pod \"930fa446-fef9-4637-965c-7a366958d102\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") "
Jul 28 22:37:29.311475 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.311330    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930fa446-fef9-4637-965c-7a366958d102-maas-api-tls\") pod \"930fa446-fef9-4637-965c-7a366958d102\" (UID: \"930fa446-fef9-4637-965c-7a366958d102\") "
Jul 28 22:37:29.311534 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.311511    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930fa446-fef9-4637-965c-7a366958d102-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "930fa446-fef9-4637-965c-7a366958d102" (UID: "930fa446-fef9-4637-965c-7a366958d102"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:37:29.311594 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.311548    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "edd8eff5-2ee7-487d-8726-aeca49a8bbbc" (UID: "edd8eff5-2ee7-487d-8726-aeca49a8bbbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:37:29.311699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.311676    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930fa446-fef9-4637-965c-7a366958d102-openshift-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:37:29.311699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.311699    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:37:29.313537 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.313514    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930fa446-fef9-4637-965c-7a366958d102-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "930fa446-fef9-4637-965c-7a366958d102" (UID: "930fa446-fef9-4637-965c-7a366958d102"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:37:29.313768 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.313751    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-kube-api-access-k862p" (OuterVolumeSpecName: "kube-api-access-k862p") pod "edd8eff5-2ee7-487d-8726-aeca49a8bbbc" (UID: "edd8eff5-2ee7-487d-8726-aeca49a8bbbc"). InnerVolumeSpecName "kube-api-access-k862p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:37:29.313768 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.313757    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930fa446-fef9-4637-965c-7a366958d102-kube-api-access-tdr5m" (OuterVolumeSpecName: "kube-api-access-tdr5m") pod "930fa446-fef9-4637-965c-7a366958d102" (UID: "930fa446-fef9-4637-965c-7a366958d102"). InnerVolumeSpecName "kube-api-access-tdr5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:37:29.412420 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.412320    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdr5m\" (UniqueName: \"kubernetes.io/projected/930fa446-fef9-4637-965c-7a366958d102-kube-api-access-tdr5m\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:37:29.412420 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.412363    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930fa446-fef9-4637-965c-7a366958d102-maas-api-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:37:29.412420 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.412373    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k862p\" (UniqueName: \"kubernetes.io/projected/edd8eff5-2ee7-487d-8726-aeca49a8bbbc-kube-api-access-k862p\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:37:29.452348 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.452306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:29.653349 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.653310    2560 generic.go:356] "Generic (PLEG): container finished" podID="930fa446-fef9-4637-965c-7a366958d102" containerID="52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d" exitCode=0
Jul 28 22:37:29.653638 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.653366    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"
Jul 28 22:37:29.653638 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.653391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x" event={"ID":"930fa446-fef9-4637-965c-7a366958d102","Type":"ContainerDied","Data":"52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d"}
Jul 28 22:37:29.653638 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.653483    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x" event={"ID":"930fa446-fef9-4637-965c-7a366958d102","Type":"ContainerDied","Data":"583e6c3daeef7697126298e18d72f96330736171efea84735e41d47c4a4beed9"}
Jul 28 22:37:29.653638 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.653516    2560 scope.go:117] "RemoveContainer" containerID="52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d"
Jul 28 22:37:29.654972 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.654950    2560 generic.go:356] "Generic (PLEG): container finished" podID="edd8eff5-2ee7-487d-8726-aeca49a8bbbc" containerID="4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac" exitCode=0
Jul 28 22:37:29.655073 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.654999    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"
Jul 28 22:37:29.655073 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.655026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh" event={"ID":"edd8eff5-2ee7-487d-8726-aeca49a8bbbc","Type":"ContainerDied","Data":"4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac"}
Jul 28 22:37:29.655222 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.655082    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh" event={"ID":"edd8eff5-2ee7-487d-8726-aeca49a8bbbc","Type":"ContainerDied","Data":"ec46b258b6d5b0a2292d8fb330eb10b3f5ba80f81def9178a47c9a9cd3e868f4"}
Jul 28 22:37:29.665652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.665636    2560 scope.go:117] "RemoveContainer" containerID="52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d"
Jul 28 22:37:29.665877 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:37:29.665859    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d\": container with ID starting with 52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d not found: ID does not exist" containerID="52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d"
Jul 28 22:37:29.665937 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.665883    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d"} err="failed to get container status \"52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d\": rpc error: code = NotFound desc = could not find container \"52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d\": container with ID starting with 52b3bda0029138bc1ffddbce53444d1da3c79b6c1605aba009c00537357fdd7d not found: ID does not exist"
Jul 28 22:37:29.665937 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.665898    2560 scope.go:117] "RemoveContainer" containerID="4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac"
Jul 28 22:37:29.674283 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.674267    2560 scope.go:117] "RemoveContainer" containerID="4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac"
Jul 28 22:37:29.674517 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:37:29.674498    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac\": container with ID starting with 4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac not found: ID does not exist" containerID="4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac"
Jul 28 22:37:29.674571 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.674523    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac"} err="failed to get container status \"4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac\": rpc error: code = NotFound desc = could not find container \"4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac\": container with ID starting with 4d146e4ed01a44b9c50ddaee2050727b0fc11cbde750f8d3355697f9d885d0ac not found: ID does not exist"
Jul 28 22:37:29.681393 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.681367    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"]
Jul 28 22:37:29.686889 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.686868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:29.688207 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.688189    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6ad9588a-94d7fc95b-59rrh"]
Jul 28 22:37:29.699023 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.699001    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"]
Jul 28 22:37:29.703956 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:29.703928    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6ad9588a-649dd46f69-tj68x"]
Jul 28 22:37:30.002516 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:30.002466    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:30.100769 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:30.100730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:30.406892 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:30.406815    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930fa446-fef9-4637-965c-7a366958d102" path="/var/lib/kubelet/pods/930fa446-fef9-4637-965c-7a366958d102/volumes"
Jul 28 22:37:30.407200 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:30.407187    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edd8eff5-2ee7-487d-8726-aeca49a8bbbc" path="/var/lib/kubelet/pods/edd8eff5-2ee7-487d-8726-aeca49a8bbbc/volumes"
Jul 28 22:37:32.250235 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.250200    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:32.349522 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.349491    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:32.369905 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.369872    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"]
Jul 28 22:37:32.370450 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.370432    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930fa446-fef9-4637-965c-7a366958d102" containerName="maas-api"
Jul 28 22:37:32.370512 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.370452    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="930fa446-fef9-4637-965c-7a366958d102" containerName="maas-api"
Jul 28 22:37:32.370512 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.370477    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd8eff5-2ee7-487d-8726-aeca49a8bbbc" containerName="payload-pre-processing"
Jul 28 22:37:32.370512 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.370483    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd8eff5-2ee7-487d-8726-aeca49a8bbbc" containerName="payload-pre-processing"
Jul 28 22:37:32.370607 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.370540    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="930fa446-fef9-4637-965c-7a366958d102" containerName="maas-api"
Jul 28 22:37:32.370607 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.370553    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd8eff5-2ee7-487d-8726-aeca49a8bbbc" containerName="payload-pre-processing"
Jul 28 22:37:32.373984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.373967    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.377659 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.377641    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ae3227ae-serving-cert\""
Jul 28 22:37:32.377780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.377666    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-89lsd\""
Jul 28 22:37:32.381819 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.381800    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"]
Jul 28 22:37:32.446832 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.446803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:32.542898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.542811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7538c460-b9e1-4d29-bff0-c04a32e6cf17-openshift-service-ca\") pod \"maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.542898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.542862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmqjn\" (UniqueName: \"kubernetes.io/projected/7538c460-b9e1-4d29-bff0-c04a32e6cf17-kube-api-access-zmqjn\") pod \"maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.542898 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.542889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7538c460-b9e1-4d29-bff0-c04a32e6cf17-maas-api-tls\") pod \"maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.554352 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.554315    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:32.643780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.643739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7538c460-b9e1-4d29-bff0-c04a32e6cf17-openshift-service-ca\") pod \"maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.643943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.643800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmqjn\" (UniqueName: \"kubernetes.io/projected/7538c460-b9e1-4d29-bff0-c04a32e6cf17-kube-api-access-zmqjn\") pod \"maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.643943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.643826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7538c460-b9e1-4d29-bff0-c04a32e6cf17-maas-api-tls\") pod \"maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.644469 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.644446    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7538c460-b9e1-4d29-bff0-c04a32e6cf17-openshift-service-ca\") pod \"maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.646277 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.646249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7538c460-b9e1-4d29-bff0-c04a32e6cf17-maas-api-tls\") pod \"maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.651295 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.651274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmqjn\" (UniqueName: \"kubernetes.io/projected/7538c460-b9e1-4d29-bff0-c04a32e6cf17-kube-api-access-zmqjn\") pod \"maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.686530 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.686501    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:32.824242 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.824217    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"]
Jul 28 22:37:32.826429 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:37:32.826398    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7538c460_b9e1_4d29_bff0_c04a32e6cf17.slice/crio-dde6f57f1f719704d0619b0e0611e78672ca905a5e78ebe4ec27183bd71d7f94 WatchSource:0}: Error finding container dde6f57f1f719704d0619b0e0611e78672ca905a5e78ebe4ec27183bd71d7f94: Status 404 returned error can't find the container with id dde6f57f1f719704d0619b0e0611e78672ca905a5e78ebe4ec27183bd71d7f94
Jul 28 22:37:32.932141 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:32.932095    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:33.024798 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:33.024770    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:37:33.677559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:33.677476    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm" event={"ID":"7538c460-b9e1-4d29-bff0-c04a32e6cf17","Type":"ContainerStarted","Data":"2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f"}
Jul 28 22:37:33.677559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:33.677527    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm" event={"ID":"7538c460-b9e1-4d29-bff0-c04a32e6cf17","Type":"ContainerStarted","Data":"dde6f57f1f719704d0619b0e0611e78672ca905a5e78ebe4ec27183bd71d7f94"}
Jul 28 22:37:33.677938 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:33.677567    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:33.696509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:33.696457    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm" podStartSLOduration=1.275747512 podStartE2EDuration="1.696439486s" totalImagesPullingTime="420.691974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:37:33.694840068 +0000 UTC m=+1655.817187983" watchObservedRunningTime="2026-07-28 22:37:33.696439486 +0000 UTC m=+1655.818787378"
Jul 28 22:37:39.687593 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:39.687556    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:37:58.057541 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.057506    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"]
Jul 28 22:37:58.061256 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.061235    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"
Jul 28 22:37:58.063739 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.063718    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 22:37:58.069917 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.069884    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"]
Jul 28 22:37:58.199529 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.199493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ed3291-1d5b-4122-814d-abf8f713458c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5\" (UID: \"47ed3291-1d5b-4122-814d-abf8f713458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"
Jul 28 22:37:58.300371 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.300314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ed3291-1d5b-4122-814d-abf8f713458c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5\" (UID: \"47ed3291-1d5b-4122-814d-abf8f713458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"
Jul 28 22:37:58.300926 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.300903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ed3291-1d5b-4122-814d-abf8f713458c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5\" (UID: \"47ed3291-1d5b-4122-814d-abf8f713458c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"
Jul 28 22:37:58.375392 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.375306    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 22:37:58.383701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.383674    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"
Jul 28 22:37:58.725675 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.725649    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"]
Jul 28 22:37:58.727482 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:37:58.727449    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ed3291_1d5b_4122_814d_abf8f713458c.slice/crio-362a7a9d9ea77fc0ccbdd74d4c801a4890d47ebc5f3149dc53f1ded0e70af772 WatchSource:0}: Error finding container 362a7a9d9ea77fc0ccbdd74d4c801a4890d47ebc5f3149dc53f1ded0e70af772: Status 404 returned error can't find the container with id 362a7a9d9ea77fc0ccbdd74d4c801a4890d47ebc5f3149dc53f1ded0e70af772
Jul 28 22:37:58.805507 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:58.802885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5" event={"ID":"47ed3291-1d5b-4122-814d-abf8f713458c","Type":"ContainerStarted","Data":"362a7a9d9ea77fc0ccbdd74d4c801a4890d47ebc5f3149dc53f1ded0e70af772"}
Jul 28 22:37:59.809120 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:59.809064    2560 generic.go:356] "Generic (PLEG): container finished" podID="47ed3291-1d5b-4122-814d-abf8f713458c" containerID="b6010ca5b056137f9d3aeaf1233949d1b942f4fb84008b7d0dd12ebbea3a83a8" exitCode=0
Jul 28 22:37:59.809474 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:37:59.809153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5" event={"ID":"47ed3291-1d5b-4122-814d-abf8f713458c","Type":"ContainerDied","Data":"b6010ca5b056137f9d3aeaf1233949d1b942f4fb84008b7d0dd12ebbea3a83a8"}
Jul 28 22:38:00.946661 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:00.946638    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"
Jul 28 22:38:01.130313 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:01.130223    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ed3291-1d5b-4122-814d-abf8f713458c-maas-api-service-ca\") pod \"47ed3291-1d5b-4122-814d-abf8f713458c\" (UID: \"47ed3291-1d5b-4122-814d-abf8f713458c\") "
Jul 28 22:38:01.130592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:01.130570    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ed3291-1d5b-4122-814d-abf8f713458c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47ed3291-1d5b-4122-814d-abf8f713458c" (UID: "47ed3291-1d5b-4122-814d-abf8f713458c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:38:01.231078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:01.231042    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ed3291-1d5b-4122-814d-abf8f713458c-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:38:01.820225 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:01.820194    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"
Jul 28 22:38:01.820377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:01.820236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5" event={"ID":"47ed3291-1d5b-4122-814d-abf8f713458c","Type":"ContainerDied","Data":"362a7a9d9ea77fc0ccbdd74d4c801a4890d47ebc5f3149dc53f1ded0e70af772"}
Jul 28 22:38:01.820377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:01.820284    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="362a7a9d9ea77fc0ccbdd74d4c801a4890d47ebc5f3149dc53f1ded0e70af772"
Jul 28 22:38:02.984693 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:02.984648    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"]
Jul 28 22:38:02.985194 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:02.984880    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm" podUID="7538c460-b9e1-4d29-bff0-c04a32e6cf17" containerName="maas-api" containerID="cri-o://2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f" gracePeriod=30
Jul 28 22:38:03.013887 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.013853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:03.080677 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.080636    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:03.140370 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.140323    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:03.184357 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.184320    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"]
Jul 28 22:38:03.186228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.186206    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae3227ae-m8ht5"]
Jul 28 22:38:03.199774 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.199726    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:03.245052 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.244989    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:38:03.350670 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.350639    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmqjn\" (UniqueName: \"kubernetes.io/projected/7538c460-b9e1-4d29-bff0-c04a32e6cf17-kube-api-access-zmqjn\") pod \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") "
Jul 28 22:38:03.350670 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.350671    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7538c460-b9e1-4d29-bff0-c04a32e6cf17-maas-api-tls\") pod \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") "
Jul 28 22:38:03.350939 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.350754    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7538c460-b9e1-4d29-bff0-c04a32e6cf17-openshift-service-ca\") pod \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\" (UID: \"7538c460-b9e1-4d29-bff0-c04a32e6cf17\") "
Jul 28 22:38:03.351136 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.351094    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7538c460-b9e1-4d29-bff0-c04a32e6cf17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7538c460-b9e1-4d29-bff0-c04a32e6cf17" (UID: "7538c460-b9e1-4d29-bff0-c04a32e6cf17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:38:03.353078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.353048    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7538c460-b9e1-4d29-bff0-c04a32e6cf17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7538c460-b9e1-4d29-bff0-c04a32e6cf17" (UID: "7538c460-b9e1-4d29-bff0-c04a32e6cf17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:38:03.353196 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.353161    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7538c460-b9e1-4d29-bff0-c04a32e6cf17-kube-api-access-zmqjn" (OuterVolumeSpecName: "kube-api-access-zmqjn") pod "7538c460-b9e1-4d29-bff0-c04a32e6cf17" (UID: "7538c460-b9e1-4d29-bff0-c04a32e6cf17"). InnerVolumeSpecName "kube-api-access-zmqjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:38:03.451984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.451954    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmqjn\" (UniqueName: \"kubernetes.io/projected/7538c460-b9e1-4d29-bff0-c04a32e6cf17-kube-api-access-zmqjn\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:38:03.451984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.451980    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7538c460-b9e1-4d29-bff0-c04a32e6cf17-maas-api-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:38:03.451984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.451990    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7538c460-b9e1-4d29-bff0-c04a32e6cf17-openshift-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:38:03.832435 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.832403    2560 generic.go:356] "Generic (PLEG): container finished" podID="7538c460-b9e1-4d29-bff0-c04a32e6cf17" containerID="2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f" exitCode=0
Jul 28 22:38:03.832617 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.832445    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"
Jul 28 22:38:03.832617 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.832496    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm" event={"ID":"7538c460-b9e1-4d29-bff0-c04a32e6cf17","Type":"ContainerDied","Data":"2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f"}
Jul 28 22:38:03.832617 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.832549    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm" event={"ID":"7538c460-b9e1-4d29-bff0-c04a32e6cf17","Type":"ContainerDied","Data":"dde6f57f1f719704d0619b0e0611e78672ca905a5e78ebe4ec27183bd71d7f94"}
Jul 28 22:38:03.832617 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.832573    2560 scope.go:117] "RemoveContainer" containerID="2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f"
Jul 28 22:38:03.842296 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.842281    2560 scope.go:117] "RemoveContainer" containerID="2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f"
Jul 28 22:38:03.842570 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:38:03.842542    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f\": container with ID starting with 2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f not found: ID does not exist" containerID="2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f"
Jul 28 22:38:03.842695 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.842571    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f"} err="failed to get container status \"2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f\": rpc error: code = NotFound desc = could not find container \"2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f\": container with ID starting with 2bc6557f1f33b11862c77e2b24b88e048261537317ba200e6ebc24b52a74e33f not found: ID does not exist"
Jul 28 22:38:03.854285 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.854262    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"]
Jul 28 22:38:03.857848 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:03.857828    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3227ae-847cb48f84-kx2qm"]
Jul 28 22:38:04.367534 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:04.367493    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:04.406701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:04.406666    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ed3291-1d5b-4122-814d-abf8f713458c" path="/var/lib/kubelet/pods/47ed3291-1d5b-4122-814d-abf8f713458c/volumes"
Jul 28 22:38:04.407085 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:04.407071    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7538c460-b9e1-4d29-bff0-c04a32e6cf17" path="/var/lib/kubelet/pods/7538c460-b9e1-4d29-bff0-c04a32e6cf17/volumes"
Jul 28 22:38:04.597517 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:04.597484    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:04.863132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:04.863078    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:05.150953 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:05.150880    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:05.386295 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:05.386262    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:07.461088 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.461058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:07.520994 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.520957    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"]
Jul 28 22:38:07.521390 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.521375    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7538c460-b9e1-4d29-bff0-c04a32e6cf17" containerName="maas-api"
Jul 28 22:38:07.521443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.521391    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7538c460-b9e1-4d29-bff0-c04a32e6cf17" containerName="maas-api"
Jul 28 22:38:07.521443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.521436    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ed3291-1d5b-4122-814d-abf8f713458c" containerName="revoke-keys"
Jul 28 22:38:07.521443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.521442    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ed3291-1d5b-4122-814d-abf8f713458c" containerName="revoke-keys"
Jul 28 22:38:07.521551 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.521505    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7538c460-b9e1-4d29-bff0-c04a32e6cf17" containerName="maas-api"
Jul 28 22:38:07.521551 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.521514    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ed3291-1d5b-4122-814d-abf8f713458c" containerName="revoke-keys"
Jul 28 22:38:07.525803 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.525779    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:07.528639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.528615    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b9e632fc-serving-cert\""
Jul 28 22:38:07.528772 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.528693    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-89lsd\""
Jul 28 22:38:07.535137 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.534510    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"]
Jul 28 22:38:07.559291 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.559198    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:07.593626 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.593592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-maas-api-tls\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:07.593807 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.593698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kq5x\" (UniqueName: \"kubernetes.io/projected/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-kube-api-access-6kq5x\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:07.593807 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.593734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-openshift-service-ca\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:07.632246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.632215    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:07.694671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.694636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kq5x\" (UniqueName: \"kubernetes.io/projected/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-kube-api-access-6kq5x\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:07.694827 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.694684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-openshift-service-ca\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:07.694827 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.694742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-maas-api-tls\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:07.694897 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:38:07.694853    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-serving-cert: secret "maas-api-e2e-derive-b9e632fc-serving-cert" not found
Jul 28 22:38:07.694931 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:38:07.694913    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-maas-api-tls podName:ad0be111-726d-43e5-af6b-ddd0c8b37c7b nodeName:}" failed. No retries permitted until 2026-07-28 22:38:08.19489472 +0000 UTC m=+1690.317242591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-maas-api-tls") pod "maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp" (UID: "ad0be111-726d-43e5-af6b-ddd0c8b37c7b") : secret "maas-api-e2e-derive-b9e632fc-serving-cert" not found
Jul 28 22:38:07.695443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.695422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-openshift-service-ca\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:07.703294 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.703268    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kq5x\" (UniqueName: \"kubernetes.io/projected/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-kube-api-access-6kq5x\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:07.930047 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:07.929997    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:08.034159 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.034087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:08.130353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.130318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:08.199602 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.199514    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-maas-api-tls\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:08.202291 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.202261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-maas-api-tls\") pod \"maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:08.203438 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.203417    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"]
Jul 28 22:38:08.207154 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.207137    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:08.209699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.209651    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b9e632fc\""
Jul 28 22:38:08.209699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.209691    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b9e632fc-dockercfg-mzqmg\""
Jul 28 22:38:08.217824 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.217804    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"]
Jul 28 22:38:08.300668 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.300628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58b4cfa-88d5-430b-825e-a82ce566fc9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g\" (UID: \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:08.300668 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.300670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxn7h\" (UniqueName: \"kubernetes.io/projected/a58b4cfa-88d5-430b-825e-a82ce566fc9f-kube-api-access-wxn7h\") pod \"payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g\" (UID: \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:08.401526 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.401487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58b4cfa-88d5-430b-825e-a82ce566fc9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g\" (UID: \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:08.401526 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.401521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxn7h\" (UniqueName: \"kubernetes.io/projected/a58b4cfa-88d5-430b-825e-a82ce566fc9f-kube-api-access-wxn7h\") pod \"payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g\" (UID: \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:08.402416 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.402389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58b4cfa-88d5-430b-825e-a82ce566fc9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g\" (UID: \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:08.416225 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.416198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxn7h\" (UniqueName: \"kubernetes.io/projected/a58b4cfa-88d5-430b-825e-a82ce566fc9f-kube-api-access-wxn7h\") pod \"payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g\" (UID: \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:08.438816 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.438777    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:08.520996 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.520958    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:08.582497 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.582462    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"]
Jul 28 22:38:08.585846 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:38:08.585811    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0be111_726d_43e5_af6b_ddd0c8b37c7b.slice/crio-f4b0d1c0b5089b884033096df087d5b84f3b8cdb211682e46ab0735dc4b01586 WatchSource:0}: Error finding container f4b0d1c0b5089b884033096df087d5b84f3b8cdb211682e46ab0735dc4b01586: Status 404 returned error can't find the container with id f4b0d1c0b5089b884033096df087d5b84f3b8cdb211682e46ab0735dc4b01586
Jul 28 22:38:08.862516 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.862429    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp" event={"ID":"ad0be111-726d-43e5-af6b-ddd0c8b37c7b","Type":"ContainerStarted","Data":"f4b0d1c0b5089b884033096df087d5b84f3b8cdb211682e46ab0735dc4b01586"}
Jul 28 22:38:08.915735 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:08.915711    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"]
Jul 28 22:38:08.918045 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:38:08.918022    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58b4cfa_88d5_430b_825e_a82ce566fc9f.slice/crio-b034414140b9066bcf1daf6be10adc99574f4432cb8571de861c39dac8ec470f WatchSource:0}: Error finding container b034414140b9066bcf1daf6be10adc99574f4432cb8571de861c39dac8ec470f: Status 404 returned error can't find the container with id b034414140b9066bcf1daf6be10adc99574f4432cb8571de861c39dac8ec470f
Jul 28 22:38:09.869473 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:09.869429    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g" event={"ID":"a58b4cfa-88d5-430b-825e-a82ce566fc9f","Type":"ContainerStarted","Data":"6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85"}
Jul 28 22:38:09.869920 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:09.869492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g" event={"ID":"a58b4cfa-88d5-430b-825e-a82ce566fc9f","Type":"ContainerStarted","Data":"b034414140b9066bcf1daf6be10adc99574f4432cb8571de861c39dac8ec470f"}
Jul 28 22:38:09.869920 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:09.869529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:09.870957 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:09.870933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp" event={"ID":"ad0be111-726d-43e5-af6b-ddd0c8b37c7b","Type":"ContainerStarted","Data":"bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef"}
Jul 28 22:38:09.871076 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:09.871061    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:09.886838 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:09.886791    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g" podStartSLOduration=1.886777613 podStartE2EDuration="1.886777613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:38:09.884712769 +0000 UTC m=+1692.007060662" watchObservedRunningTime="2026-07-28 22:38:09.886777613 +0000 UTC m=+1692.009125506"
Jul 28 22:38:09.900640 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:09.900592    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp" podStartSLOduration=2.593351433 podStartE2EDuration="2.900574928s" totalImagesPullingTime="307.223495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:38:09.899743926 +0000 UTC m=+1692.022091841" watchObservedRunningTime="2026-07-28 22:38:09.900574928 +0000 UTC m=+1692.022922833"
Jul 28 22:38:15.883344 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:15.883308    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:20.878328 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:20.878247    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:37.441209 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.441176    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"]
Jul 28 22:38:37.441885 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.441431    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp" podUID="ad0be111-726d-43e5-af6b-ddd0c8b37c7b" containerName="maas-api" containerID="cri-o://bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef" gracePeriod=30
Jul 28 22:38:37.473255 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.473227    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:37.503895 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.503717    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"]
Jul 28 22:38:37.504070 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.504006    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g" podUID="a58b4cfa-88d5-430b-825e-a82ce566fc9f" containerName="payload-pre-processing" containerID="cri-o://6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85" gracePeriod=30
Jul 28 22:38:37.571458 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.571421    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:37.673916 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.673878    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:37.746708 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.746680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:37.753323 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.753302    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:37.778090 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.778067    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:37.888347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.888315    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:37.892436 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.892411    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58b4cfa-88d5-430b-825e-a82ce566fc9f-plugins-config-volume\") pod \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\" (UID: \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\") "
Jul 28 22:38:37.892577 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.892471    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-openshift-service-ca\") pod \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") "
Jul 28 22:38:37.892577 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.892500    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-maas-api-tls\") pod \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") "
Jul 28 22:38:37.892577 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.892544    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kq5x\" (UniqueName: \"kubernetes.io/projected/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-kube-api-access-6kq5x\") pod \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\" (UID: \"ad0be111-726d-43e5-af6b-ddd0c8b37c7b\") "
Jul 28 22:38:37.892577 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.892561    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxn7h\" (UniqueName: \"kubernetes.io/projected/a58b4cfa-88d5-430b-825e-a82ce566fc9f-kube-api-access-wxn7h\") pod \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\" (UID: \"a58b4cfa-88d5-430b-825e-a82ce566fc9f\") "
Jul 28 22:38:37.892841 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.892815    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad0be111-726d-43e5-af6b-ddd0c8b37c7b" (UID: "ad0be111-726d-43e5-af6b-ddd0c8b37c7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:38:37.892906 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.892855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58b4cfa-88d5-430b-825e-a82ce566fc9f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a58b4cfa-88d5-430b-825e-a82ce566fc9f" (UID: "a58b4cfa-88d5-430b-825e-a82ce566fc9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:38:37.894712 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.894679    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-kube-api-access-6kq5x" (OuterVolumeSpecName: "kube-api-access-6kq5x") pod "ad0be111-726d-43e5-af6b-ddd0c8b37c7b" (UID: "ad0be111-726d-43e5-af6b-ddd0c8b37c7b"). InnerVolumeSpecName "kube-api-access-6kq5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:38:37.894825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.894709    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58b4cfa-88d5-430b-825e-a82ce566fc9f-kube-api-access-wxn7h" (OuterVolumeSpecName: "kube-api-access-wxn7h") pod "a58b4cfa-88d5-430b-825e-a82ce566fc9f" (UID: "a58b4cfa-88d5-430b-825e-a82ce566fc9f"). InnerVolumeSpecName "kube-api-access-wxn7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:38:37.894825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.894781    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad0be111-726d-43e5-af6b-ddd0c8b37c7b" (UID: "ad0be111-726d-43e5-af6b-ddd0c8b37c7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:38:37.993436 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.993408    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxn7h\" (UniqueName: \"kubernetes.io/projected/a58b4cfa-88d5-430b-825e-a82ce566fc9f-kube-api-access-wxn7h\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:38:37.993436 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.993436    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58b4cfa-88d5-430b-825e-a82ce566fc9f-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:38:37.993633 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.993448    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-openshift-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:38:37.993633 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.993458    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-maas-api-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:38:37.993633 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:37.993468    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kq5x\" (UniqueName: \"kubernetes.io/projected/ad0be111-726d-43e5-af6b-ddd0c8b37c7b-kube-api-access-6kq5x\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:38:38.020073 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.020042    2560 generic.go:356] "Generic (PLEG): container finished" podID="ad0be111-726d-43e5-af6b-ddd0c8b37c7b" containerID="bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef" exitCode=0
Jul 28 22:38:38.020251 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.020088    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"
Jul 28 22:38:38.020251 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.020128    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp" event={"ID":"ad0be111-726d-43e5-af6b-ddd0c8b37c7b","Type":"ContainerDied","Data":"bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef"}
Jul 28 22:38:38.020251 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.020183    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp" event={"ID":"ad0be111-726d-43e5-af6b-ddd0c8b37c7b","Type":"ContainerDied","Data":"f4b0d1c0b5089b884033096df087d5b84f3b8cdb211682e46ab0735dc4b01586"}
Jul 28 22:38:38.020251 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.020206    2560 scope.go:117] "RemoveContainer" containerID="bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef"
Jul 28 22:38:38.021641 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.021618    2560 generic.go:356] "Generic (PLEG): container finished" podID="a58b4cfa-88d5-430b-825e-a82ce566fc9f" containerID="6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85" exitCode=0
Jul 28 22:38:38.021735 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.021657    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"
Jul 28 22:38:38.021735 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.021682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g" event={"ID":"a58b4cfa-88d5-430b-825e-a82ce566fc9f","Type":"ContainerDied","Data":"6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85"}
Jul 28 22:38:38.021735 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.021721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g" event={"ID":"a58b4cfa-88d5-430b-825e-a82ce566fc9f","Type":"ContainerDied","Data":"b034414140b9066bcf1daf6be10adc99574f4432cb8571de861c39dac8ec470f"}
Jul 28 22:38:38.029680 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.029665    2560 scope.go:117] "RemoveContainer" containerID="bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef"
Jul 28 22:38:38.029940 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:38:38.029923    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef\": container with ID starting with bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef not found: ID does not exist" containerID="bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef"
Jul 28 22:38:38.030003 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.029947    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef"} err="failed to get container status \"bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef\": rpc error: code = NotFound desc = could not find container \"bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef\": container with ID starting with bc8cc2ffb83a8247aefe7123062e383abcd5a689bbd86c6c50595fe50d7d1bef not found: ID does not exist"
Jul 28 22:38:38.030003 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.029962    2560 scope.go:117] "RemoveContainer" containerID="6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85"
Jul 28 22:38:38.040143 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.040120    2560 scope.go:117] "RemoveContainer" containerID="6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85"
Jul 28 22:38:38.040402 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:38:38.040386    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85\": container with ID starting with 6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85 not found: ID does not exist" containerID="6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85"
Jul 28 22:38:38.040449 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.040409    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85"} err="failed to get container status \"6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85\": rpc error: code = NotFound desc = could not find container \"6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85\": container with ID starting with 6c91bcddd25394102c9293b44bd00ce8f14e7dc252322ed2b97d39a86fc6ec85 not found: ID does not exist"
Jul 28 22:38:38.044962 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.044942    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"]
Jul 28 22:38:38.050183 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.050163    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9e632fc-58d55479f9-4vcbp"]
Jul 28 22:38:38.060188 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.060168    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"]
Jul 28 22:38:38.067070 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.067050    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9e632fc-84d649f988-dv92g"]
Jul 28 22:38:38.108701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.108671    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:38.406836 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.406756    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58b4cfa-88d5-430b-825e-a82ce566fc9f" path="/var/lib/kubelet/pods/a58b4cfa-88d5-430b-825e-a82ce566fc9f/volumes"
Jul 28 22:38:38.407167 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.407153    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0be111-726d-43e5-af6b-ddd0c8b37c7b" path="/var/lib/kubelet/pods/ad0be111-726d-43e5-af6b-ddd0c8b37c7b/volumes"
Jul 28 22:38:38.577205 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.577172    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:38.677804 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.677708    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:38.781024 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:38.780991    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:40.177374 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:40.177345    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:46.416287 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:46.416252    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:46.520540 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:46.520504    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:46.618512 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:46.618472    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:46.717510 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:46.717484    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:38:47.188651 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:38:47.188563    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:00.242982 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:00.242943    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:13.139767 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:13.139732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:19.187306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:19.187267    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:19.293066 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:19.293031    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:19.405094 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:19.405060    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:35.310557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:35.310519    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:40.927143 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:40.927089    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:40.982504 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:40.982472    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:41.074004 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:41.073959    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:41.167910 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:41.167872    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:51.649733 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:51.649644    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:52.433611 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:52.433577    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:58.554379 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:58.554344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:58.661739 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:58.661709    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:58.751669 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:58.751635    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:39:58.851830 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:39:58.851737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:10.348746 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:10.348710    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:10.478910 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:10.478874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:10.600146 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:10.600038    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:12.287673 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:12.287635    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:17.907743 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:17.907699    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:17.969950 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:17.969911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:18.605455 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:18.605419    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:24.465219 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:24.465182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:24.525990 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:24.525956    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:24.703987 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:24.703951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:25.068083 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:25.068039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:25.301137 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:25.301082    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:27.799481 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:27.799443    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:27.875026 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:27.874978    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:27.959219 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:27.959179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:28.086943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:28.086849    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:28.620056 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:28.620011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:28.712619 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:28.712581    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:52.792083 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:52.792041    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:54.255405 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:54.255368    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:59.673767 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:59.673728    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:40:59.730391 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:40:59.730361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:00.056476 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.056439    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"]
Jul 28 22:41:00.056868 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.056854    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58b4cfa-88d5-430b-825e-a82ce566fc9f" containerName="payload-pre-processing"
Jul 28 22:41:00.056914 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.056869    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58b4cfa-88d5-430b-825e-a82ce566fc9f" containerName="payload-pre-processing"
Jul 28 22:41:00.056914 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.056902    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0be111-726d-43e5-af6b-ddd0c8b37c7b" containerName="maas-api"
Jul 28 22:41:00.056914 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.056909    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0be111-726d-43e5-af6b-ddd0c8b37c7b" containerName="maas-api"
Jul 28 22:41:00.057010 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.056981    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58b4cfa-88d5-430b-825e-a82ce566fc9f" containerName="payload-pre-processing"
Jul 28 22:41:00.057010 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.056995    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0be111-726d-43e5-af6b-ddd0c8b37c7b" containerName="maas-api"
Jul 28 22:41:00.060080 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.060064    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"
Jul 28 22:41:00.062844 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.062820    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 22:41:00.069295 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.069272    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"]
Jul 28 22:41:00.165511 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.165472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s\" (UID: \"9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"
Jul 28 22:41:00.267078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.267042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s\" (UID: \"9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"
Jul 28 22:41:00.267663 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.267643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s\" (UID: \"9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"
Jul 28 22:41:00.371532 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.371442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"
Jul 28 22:41:00.504950 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.504920    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"]
Jul 28 22:41:00.507204 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:41:00.507176    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3c4406_c0f8_4f8d_a4c1_7900ea7829e7.slice/crio-4afd58968e00467e6222eff52e7b6cc1ef6266f55af1843805c3d5f8de76c523 WatchSource:0}: Error finding container 4afd58968e00467e6222eff52e7b6cc1ef6266f55af1843805c3d5f8de76c523: Status 404 returned error can't find the container with id 4afd58968e00467e6222eff52e7b6cc1ef6266f55af1843805c3d5f8de76c523
Jul 28 22:41:00.744414 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:00.744376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s" event={"ID":"9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7","Type":"ContainerStarted","Data":"4afd58968e00467e6222eff52e7b6cc1ef6266f55af1843805c3d5f8de76c523"}
Jul 28 22:41:01.750993 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:01.750953    2560 generic.go:356] "Generic (PLEG): container finished" podID="9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7" containerID="560be0b24946d0f485fab9c2b288c851b69acf7f690ffe9fc9894806dc11dc27" exitCode=0
Jul 28 22:41:01.751427 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:01.751040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s" event={"ID":"9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7","Type":"ContainerDied","Data":"560be0b24946d0f485fab9c2b288c851b69acf7f690ffe9fc9894806dc11dc27"}
Jul 28 22:41:02.890084 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:02.890060    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"
Jul 28 22:41:02.892599 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:02.892582    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7-maas-api-service-ca\") pod \"9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7\" (UID: \"9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7\") "
Jul 28 22:41:02.892894 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:02.892875    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7" (UID: "9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:41:02.993216 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:02.993166    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:41:03.761559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:03.761530    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"
Jul 28 22:41:03.761559 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:03.761543    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s" event={"ID":"9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7","Type":"ContainerDied","Data":"4afd58968e00467e6222eff52e7b6cc1ef6266f55af1843805c3d5f8de76c523"}
Jul 28 22:41:03.761763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:03.761586    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4afd58968e00467e6222eff52e7b6cc1ef6266f55af1843805c3d5f8de76c523"
Jul 28 22:41:10.495550 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:10.495505    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:10.564232 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:10.564196    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:10.629203 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:10.629170    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:10.673994 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:10.673955    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"]
Jul 28 22:41:10.681285 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:10.681260    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b45d9e9d-2wm4s"]
Jul 28 22:41:10.698871 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:10.698841    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:11.085253 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:11.085218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:12.407587 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:12.407551    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7" path="/var/lib/kubelet/pods/9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7/volumes"
Jul 28 22:41:16.781574 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:16.781536    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:16.845349 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:16.845309    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:17.100166 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:17.100056    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:17.330353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:17.330318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:17.473764 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:17.473724    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:17.571310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:17.571272    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:17.671146 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:17.671081    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:23.368692 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:23.368598    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:23.476064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:23.476026    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:23.569435 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:23.569403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:23.734705 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:23.734656    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:29.365251 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:29.365191    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:29.465433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:29.465398    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:36.528065 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:36.527986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:36.850499 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:36.850407    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:37.115010 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:37.114920    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:40.181662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:40.181625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:40.282500 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:40.282468    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:40.416162 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:40.416124    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:40.755357 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:40.755309    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:41.037321 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.037233    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:41.135537 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.135501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:41:41.143672 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.143645    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"]
Jul 28 22:41:41.144066 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.144053    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7" containerName="revoke-keys"
Jul 28 22:41:41.144124 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.144067    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7" containerName="revoke-keys"
Jul 28 22:41:41.144169 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.144160    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d3c4406-c0f8-4f8d-a4c1-7900ea7829e7" containerName="revoke-keys"
Jul 28 22:41:41.152409 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.152383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:41:41.155463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.155439    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8f322d7b\""
Jul 28 22:41:41.155782 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.155763    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8f322d7b-dockercfg-846qq\""
Jul 28 22:41:41.157030 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.156999    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"]
Jul 28 22:41:41.157415 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.157395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg8ht\" (UniqueName: \"kubernetes.io/projected/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-kube-api-access-fg8ht\") pod \"payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv\" (UID: \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:41:41.157509 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.157430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv\" (UID: \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:41:41.258295 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.258261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg8ht\" (UniqueName: \"kubernetes.io/projected/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-kube-api-access-fg8ht\") pod \"payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv\" (UID: \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:41:41.258669 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.258303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv\" (UID: \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:41:41.259010 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.258990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv\" (UID: \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:41:41.268891 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.268860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg8ht\" (UniqueName: \"kubernetes.io/projected/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-kube-api-access-fg8ht\") pod \"payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv\" (UID: \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:41:41.464247 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.464165    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:41:41.802867 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.802840    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"]
Jul 28 22:41:41.804521 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:41:41.804492    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b4bd4c_8513_4ee6_b1a0_d0860a3d2296.slice/crio-c8d97cb9383ed4c5269762fb4730d5b42abb6c73e696b3a1ecc56ce02aae2077 WatchSource:0}: Error finding container c8d97cb9383ed4c5269762fb4730d5b42abb6c73e696b3a1ecc56ce02aae2077: Status 404 returned error can't find the container with id c8d97cb9383ed4c5269762fb4730d5b42abb6c73e696b3a1ecc56ce02aae2077
Jul 28 22:41:41.949094 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.949060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv" event={"ID":"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296","Type":"ContainerStarted","Data":"9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953"}
Jul 28 22:41:41.949344 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.949205    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:41:41.949344 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.949236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv" event={"ID":"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296","Type":"ContainerStarted","Data":"c8d97cb9383ed4c5269762fb4730d5b42abb6c73e696b3a1ecc56ce02aae2077"}
Jul 28 22:41:41.964253 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:41.964194    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv" podStartSLOduration=0.964174446 podStartE2EDuration="964.174446ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:41:41.963940783 +0000 UTC m=+1904.086288689" watchObservedRunningTime="2026-07-28 22:41:41.964174446 +0000 UTC m=+1904.086522340"
Jul 28 22:41:52.959505 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:41:52.959471    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:42:05.235562 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:05.235522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:42:16.461490 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.461455    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"]
Jul 28 22:42:16.465311 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.465290    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.467968 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.467946    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8f322d7b\"/\"default-dockercfg-j2wjm\""
Jul 28 22:42:16.469164 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.469140    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8f322d7b\"/\"openshift-service-ca.crt\""
Jul 28 22:42:16.469290 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.469222    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8f322d7b\"/\"e2e-lifecycle-model-8f322d7b-kserve-self-signed-certs\""
Jul 28 22:42:16.469290 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.469249    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8f322d7b\"/\"kube-root-ca.crt\""
Jul 28 22:42:16.474968 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.474948    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"]
Jul 28 22:42:16.499925 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.499895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:42:16.584090 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.584057    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779a0f1e-89d8-4c08-9d45-caeb954f7505-tls-certs\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.584090 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.584090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-home\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.584365 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.584187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-tmp-dir\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.584365 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.584287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpk9t\" (UniqueName: \"kubernetes.io/projected/779a0f1e-89d8-4c08-9d45-caeb954f7505-kube-api-access-lpk9t\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.584365 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.584347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-model-cache\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.584477 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.584392    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-dshm\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.685313 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.685282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpk9t\" (UniqueName: \"kubernetes.io/projected/779a0f1e-89d8-4c08-9d45-caeb954f7505-kube-api-access-lpk9t\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.685496 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.685336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-model-cache\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.685496 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.685374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-dshm\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.685496 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.685430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779a0f1e-89d8-4c08-9d45-caeb954f7505-tls-certs\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.685496 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.685453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-home\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.685496 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.685492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-tmp-dir\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.685921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.685882    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-home\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.686042 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.685963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-tmp-dir\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.686042 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.686004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-model-cache\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.687676 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.687648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-dshm\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.687814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.687798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779a0f1e-89d8-4c08-9d45-caeb954f7505-tls-certs\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.693419 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.693395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpk9t\" (UniqueName: \"kubernetes.io/projected/779a0f1e-89d8-4c08-9d45-caeb954f7505-kube-api-access-lpk9t\") pod \"e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") " pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.777779 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.777737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:16.913801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:16.913773    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"]
Jul 28 22:42:17.130763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:17.130669    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl" event={"ID":"779a0f1e-89d8-4c08-9d45-caeb954f7505","Type":"ContainerStarted","Data":"aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5"}
Jul 28 22:42:17.130763 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:17.130710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl" event={"ID":"779a0f1e-89d8-4c08-9d45-caeb954f7505","Type":"ContainerStarted","Data":"44a8d009f4584aa037d5a3d04d6f83af9113072daec20292010f8ca5858a2e29"}
Jul 28 22:42:17.150525 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:17.150478    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl" podStartSLOduration=1.150461604 podStartE2EDuration="1.150461604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:42:17.149612481 +0000 UTC m=+1939.271960410" watchObservedRunningTime="2026-07-28 22:42:17.150461604 +0000 UTC m=+1939.272809496"
Jul 28 22:42:26.778767 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:26.778731    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:26.779349 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:26.778781    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:26.792005 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:26.791969    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:27.194009 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:27.193918    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:42:31.248664 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:31.248625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:42:58.551989 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:42:58.551907    2560 scope.go:117] "RemoveContainer" containerID="d211253ec770a7b16561101cd51215692ca8ae85914de90a2aa0b3208d270c5f"
Jul 28 22:43:00.170453 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:00.170419    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:05.412521 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.412443    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:05.446917 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.446882    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"]
Jul 28 22:43:05.447190 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.447157    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv" podUID="e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296" containerName="payload-pre-processing" containerID="cri-o://9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953" gracePeriod=30
Jul 28 22:43:05.513062 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.513025    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:05.586987 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.586954    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:05.639911 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.639868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:05.701588 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.701562    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:43:05.781186 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.781153    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg8ht\" (UniqueName: \"kubernetes.io/projected/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-kube-api-access-fg8ht\") pod \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\" (UID: \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\") "
Jul 28 22:43:05.781353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.781204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-plugins-config-volume\") pod \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\" (UID: \"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296\") "
Jul 28 22:43:05.781674 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.781642    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296" (UID: "e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:43:05.783325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.783305    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-kube-api-access-fg8ht" (OuterVolumeSpecName: "kube-api-access-fg8ht") pod "e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296" (UID: "e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296"). InnerVolumeSpecName "kube-api-access-fg8ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:43:05.882451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.882415    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg8ht\" (UniqueName: \"kubernetes.io/projected/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-kube-api-access-fg8ht\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:43:05.882451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:05.882447    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:43:06.382022 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:06.381985    2560 generic.go:356] "Generic (PLEG): container finished" podID="e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296" containerID="9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953" exitCode=0
Jul 28 22:43:06.382022 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:06.382026    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"
Jul 28 22:43:06.382243 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:06.382074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv" event={"ID":"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296","Type":"ContainerDied","Data":"9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953"}
Jul 28 22:43:06.382243 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:06.382145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv" event={"ID":"e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296","Type":"ContainerDied","Data":"c8d97cb9383ed4c5269762fb4730d5b42abb6c73e696b3a1ecc56ce02aae2077"}
Jul 28 22:43:06.382243 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:06.382171    2560 scope.go:117] "RemoveContainer" containerID="9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953"
Jul 28 22:43:06.394057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:06.394037    2560 scope.go:117] "RemoveContainer" containerID="9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953"
Jul 28 22:43:06.394366 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:43:06.394348    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953\": container with ID starting with 9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953 not found: ID does not exist" containerID="9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953"
Jul 28 22:43:06.394431 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:06.394375    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953"} err="failed to get container status \"9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953\": rpc error: code = NotFound desc = could not find container \"9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953\": container with ID starting with 9c0eed3c35e1c1df3d8d9178f594d77b6c3ce1ea98879fd8cd78995c5b1ca953 not found: ID does not exist"
Jul 28 22:43:06.408262 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:06.408241    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"]
Jul 28 22:43:06.413155 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:06.413135    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f322d7b-fb5849fb-vlstv"]
Jul 28 22:43:08.407161 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:08.407123    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296" path="/var/lib/kubelet/pods/e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296/volumes"
Jul 28 22:43:08.807621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:08.807589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:13.790876 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:13.790839    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"]
Jul 28 22:43:13.829583 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:13.829550    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:27.992440 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:27.992401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:28.066388 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:28.066356    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:28.810541 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:28.810474    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl" podUID="779a0f1e-89d8-4c08-9d45-caeb954f7505" containerName="main" containerID="cri-o://aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5" gracePeriod=45
Jul 28 22:43:29.060967 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.060910    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:43:29.209328 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209292    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-tmp-dir\") pod \"779a0f1e-89d8-4c08-9d45-caeb954f7505\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") "
Jul 28 22:43:29.209328 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209330    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779a0f1e-89d8-4c08-9d45-caeb954f7505-tls-certs\") pod \"779a0f1e-89d8-4c08-9d45-caeb954f7505\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") "
Jul 28 22:43:29.209592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209350    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-model-cache\") pod \"779a0f1e-89d8-4c08-9d45-caeb954f7505\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") "
Jul 28 22:43:29.209592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-home\") pod \"779a0f1e-89d8-4c08-9d45-caeb954f7505\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") "
Jul 28 22:43:29.209592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209490    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-dshm\") pod \"779a0f1e-89d8-4c08-9d45-caeb954f7505\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") "
Jul 28 22:43:29.209592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpk9t\" (UniqueName: \"kubernetes.io/projected/779a0f1e-89d8-4c08-9d45-caeb954f7505-kube-api-access-lpk9t\") pod \"779a0f1e-89d8-4c08-9d45-caeb954f7505\" (UID: \"779a0f1e-89d8-4c08-9d45-caeb954f7505\") "
Jul 28 22:43:29.209783 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "779a0f1e-89d8-4c08-9d45-caeb954f7505" (UID: "779a0f1e-89d8-4c08-9d45-caeb954f7505"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:43:29.209783 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209652    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-model-cache" (OuterVolumeSpecName: "model-cache") pod "779a0f1e-89d8-4c08-9d45-caeb954f7505" (UID: "779a0f1e-89d8-4c08-9d45-caeb954f7505"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:43:29.209783 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209748    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-home" (OuterVolumeSpecName: "home") pod "779a0f1e-89d8-4c08-9d45-caeb954f7505" (UID: "779a0f1e-89d8-4c08-9d45-caeb954f7505"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:43:29.209899 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209883    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:43:29.209943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209900    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:43:29.209943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.209910    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:43:29.211663 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.211631    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-dshm" (OuterVolumeSpecName: "dshm") pod "779a0f1e-89d8-4c08-9d45-caeb954f7505" (UID: "779a0f1e-89d8-4c08-9d45-caeb954f7505"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:43:29.211758 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.211702    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779a0f1e-89d8-4c08-9d45-caeb954f7505-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "779a0f1e-89d8-4c08-9d45-caeb954f7505" (UID: "779a0f1e-89d8-4c08-9d45-caeb954f7505"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:43:29.211758 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.211724    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779a0f1e-89d8-4c08-9d45-caeb954f7505-kube-api-access-lpk9t" (OuterVolumeSpecName: "kube-api-access-lpk9t") pod "779a0f1e-89d8-4c08-9d45-caeb954f7505" (UID: "779a0f1e-89d8-4c08-9d45-caeb954f7505"). InnerVolumeSpecName "kube-api-access-lpk9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:43:29.311268 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.311182    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779a0f1e-89d8-4c08-9d45-caeb954f7505-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:43:29.311268 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.311212    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpk9t\" (UniqueName: \"kubernetes.io/projected/779a0f1e-89d8-4c08-9d45-caeb954f7505-kube-api-access-lpk9t\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:43:29.311268 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.311222    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779a0f1e-89d8-4c08-9d45-caeb954f7505-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:43:29.507980 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.507947    2560 generic.go:356] "Generic (PLEG): container finished" podID="779a0f1e-89d8-4c08-9d45-caeb954f7505" containerID="aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5" exitCode=0
Jul 28 22:43:29.508174 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.507986    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"
Jul 28 22:43:29.508174 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.508005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl" event={"ID":"779a0f1e-89d8-4c08-9d45-caeb954f7505","Type":"ContainerDied","Data":"aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5"}
Jul 28 22:43:29.508174 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.508035    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl" event={"ID":"779a0f1e-89d8-4c08-9d45-caeb954f7505","Type":"ContainerDied","Data":"44a8d009f4584aa037d5a3d04d6f83af9113072daec20292010f8ca5858a2e29"}
Jul 28 22:43:29.508174 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.508052    2560 scope.go:117] "RemoveContainer" containerID="aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5"
Jul 28 22:43:29.517199 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.517180    2560 scope.go:117] "RemoveContainer" containerID="aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5"
Jul 28 22:43:29.517464 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:43:29.517446    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5\": container with ID starting with aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5 not found: ID does not exist" containerID="aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5"
Jul 28 22:43:29.517503 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.517475    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5"} err="failed to get container status \"aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5\": rpc error: code = NotFound desc = could not find container \"aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5\": container with ID starting with aa7f0740fde62d40d4d346f90598d682862b9e8e06f91601523c3d214e9cb2a5 not found: ID does not exist"
Jul 28 22:43:29.529738 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.529716    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"]
Jul 28 22:43:29.531614 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:29.531594    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8f322d7b/e2e-lifecycle-model-8f322d7b-kserve-6d585f47d7-j2swl"]
Jul 28 22:43:30.408771 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:30.408738    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779a0f1e-89d8-4c08-9d45-caeb954f7505" path="/var/lib/kubelet/pods/779a0f1e-89d8-4c08-9d45-caeb954f7505/volumes"
Jul 28 22:43:50.528780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:50.528746    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:51.045889 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:51.045848    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:56.545411 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:56.545374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:56.647447 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:56.647417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:56.811456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:56.811365    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:57.074851 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:57.074769    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:58.151583 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:58.151550    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:58.250869 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:58.250831    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:58.351651 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:58.351614    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:59.102975 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:59.102937    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:59.204143 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:59.204076    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:43:59.301783 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:43:59.301745    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:00.690116 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:00.690071    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:06.441393 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:06.441353    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:06.501144 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:06.501085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:07.136900 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:07.136863    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:13.050218 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:13.050182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:13.161548 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:13.161509    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:13.250766 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:13.250721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:23.781331 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:23.781247    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:24.122301 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:24.122214    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:45.130048 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:45.130017    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:51.567626 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:51.567580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:51.633613 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:51.633571    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:51.931847 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:51.931762    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:52.261259 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:52.261220    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:52.358372 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:52.358324    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:55.347274 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:55.347237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:55.435456 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:55.435405    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:55.501269 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:55.501221    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:55.688966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:55.688876    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:56.119512 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.119476    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:56.229385 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.229348    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:44:56.253627 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.253597    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"]
Jul 28 22:44:56.254085 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.254068    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296" containerName="payload-pre-processing"
Jul 28 22:44:56.254157 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.254087    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296" containerName="payload-pre-processing"
Jul 28 22:44:56.254157 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.254117    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779a0f1e-89d8-4c08-9d45-caeb954f7505" containerName="main"
Jul 28 22:44:56.254157 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.254124    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="779a0f1e-89d8-4c08-9d45-caeb954f7505" containerName="main"
Jul 28 22:44:56.254252 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.254193    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="779a0f1e-89d8-4c08-9d45-caeb954f7505" containerName="main"
Jul 28 22:44:56.254252 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.254202    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b4bd4c-8513-4ee6-b1a0-d0860a3d2296" containerName="payload-pre-processing"
Jul 28 22:44:56.257403 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.257382    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:44:56.260149 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.260131    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-104d62-dockercfg-g4fr2\""
Jul 28 22:44:56.260377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.260360    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-104d62\""
Jul 28 22:44:56.267639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.267618    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"]
Jul 28 22:44:56.325764 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.325730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkb8z\" (UniqueName: \"kubernetes.io/projected/c3013e3a-138d-4966-97e1-4a875b1d4c10-kube-api-access-rkb8z\") pod \"payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl\" (UID: \"c3013e3a-138d-4966-97e1-4a875b1d4c10\") " pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:44:56.325939 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.325805    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3013e3a-138d-4966-97e1-4a875b1d4c10-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl\" (UID: \"c3013e3a-138d-4966-97e1-4a875b1d4c10\") " pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:44:56.426751 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.426661    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3013e3a-138d-4966-97e1-4a875b1d4c10-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl\" (UID: \"c3013e3a-138d-4966-97e1-4a875b1d4c10\") " pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:44:56.427228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.427092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkb8z\" (UniqueName: \"kubernetes.io/projected/c3013e3a-138d-4966-97e1-4a875b1d4c10-kube-api-access-rkb8z\") pod \"payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl\" (UID: \"c3013e3a-138d-4966-97e1-4a875b1d4c10\") " pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:44:56.427524 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.427502    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3013e3a-138d-4966-97e1-4a875b1d4c10-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl\" (UID: \"c3013e3a-138d-4966-97e1-4a875b1d4c10\") " pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:44:56.435829 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.435805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkb8z\" (UniqueName: \"kubernetes.io/projected/c3013e3a-138d-4966-97e1-4a875b1d4c10-kube-api-access-rkb8z\") pod \"payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl\" (UID: \"c3013e3a-138d-4966-97e1-4a875b1d4c10\") " pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:44:56.569965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.569930    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:44:56.700594 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.700569    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"]
Jul 28 22:44:56.702353 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:44:56.702327    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3013e3a_138d_4966_97e1_4a875b1d4c10.slice/crio-02b52ef274380305c5a99038f708a5083597869a0712c431e75c5bb554e98e4b WatchSource:0}: Error finding container 02b52ef274380305c5a99038f708a5083597869a0712c431e75c5bb554e98e4b: Status 404 returned error can't find the container with id 02b52ef274380305c5a99038f708a5083597869a0712c431e75c5bb554e98e4b
Jul 28 22:44:56.964019 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.963930    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl" event={"ID":"c3013e3a-138d-4966-97e1-4a875b1d4c10","Type":"ContainerStarted","Data":"b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f"}
Jul 28 22:44:56.964019 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.963982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl" event={"ID":"c3013e3a-138d-4966-97e1-4a875b1d4c10","Type":"ContainerStarted","Data":"02b52ef274380305c5a99038f708a5083597869a0712c431e75c5bb554e98e4b"}
Jul 28 22:44:56.964272 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.964069    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:44:56.982936 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:56.982873    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl" podStartSLOduration=0.982856785 podStartE2EDuration="982.856785ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:56.981477311 +0000 UTC m=+2099.103825222" watchObservedRunningTime="2026-07-28 22:44:56.982856785 +0000 UTC m=+2099.105204679"
Jul 28 22:44:58.587721 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:44:58.587690    2560 scope.go:117] "RemoveContainer" containerID="b6010ca5b056137f9d3aeaf1233949d1b942f4fb84008b7d0dd12ebbea3a83a8"
Jul 28 22:45:00.127793 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.127759    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"]
Jul 28 22:45:00.132974 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.132951    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h"]
Jul 28 22:45:00.133123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.133065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"
Jul 28 22:45:00.135875 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.135851    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 22:45:00.136465 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.136433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h"
Jul 28 22:45:00.138208 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.138192    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"]
Jul 28 22:45:00.140802 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.140782    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h"]
Jul 28 22:45:00.147050 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.147028    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"
Jul 28 22:45:00.154188 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.154172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h"
Jul 28 22:45:00.294513 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.294476    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"]
Jul 28 22:45:00.298066 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:45:00.298038    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdae04d_6b5a_4f08_856e_0180f1ac1ee3.slice/crio-d2f65b34d17a5a302f0034eedbb10e8168823b614397e855e022b2479298178c WatchSource:0}: Error finding container d2f65b34d17a5a302f0034eedbb10e8168823b614397e855e022b2479298178c: Status 404 returned error can't find the container with id d2f65b34d17a5a302f0034eedbb10e8168823b614397e855e022b2479298178c
Jul 28 22:45:00.310992 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.310970    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h"]
Jul 28 22:45:00.312772 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:45:00.312749    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac4035b_a9a7_413e_8fd2_20027d12f074.slice/crio-40e3282479a6547b7ce2e9f115aaa292093a6224ebfec765e9fdf44a62ac1396 WatchSource:0}: Error finding container 40e3282479a6547b7ce2e9f115aaa292093a6224ebfec765e9fdf44a62ac1396: Status 404 returned error can't find the container with id 40e3282479a6547b7ce2e9f115aaa292093a6224ebfec765e9fdf44a62ac1396
Jul 28 22:45:00.985711 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.985679    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h" event={"ID":"2ac4035b-a9a7-413e-8fd2-20027d12f074","Type":"ContainerStarted","Data":"de54fb45ce7112ca9066596d5afaa46b1f7316860e589fc92cb434a083f47745"}
Jul 28 22:45:00.985901 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.985733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h" event={"ID":"2ac4035b-a9a7-413e-8fd2-20027d12f074","Type":"ContainerStarted","Data":"40e3282479a6547b7ce2e9f115aaa292093a6224ebfec765e9fdf44a62ac1396"}
Jul 28 22:45:00.987370 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.987336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" event={"ID":"bfdae04d-6b5a-4f08-856e-0180f1ac1ee3","Type":"ContainerStarted","Data":"51fa1f30eacccd51fe9c39342c3987aff8aeed256a8feac5d9e5f39931cc6043"}
Jul 28 22:45:00.987500 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:00.987386    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" event={"ID":"bfdae04d-6b5a-4f08-856e-0180f1ac1ee3","Type":"ContainerStarted","Data":"d2f65b34d17a5a302f0034eedbb10e8168823b614397e855e022b2479298178c"}
Jul 28 22:45:01.000132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:01.000011    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h" podStartSLOduration=0.999991612 podStartE2EDuration="999.991612ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:45:00.999366591 +0000 UTC m=+2103.121714511" watchObservedRunningTime="2026-07-28 22:45:00.999991612 +0000 UTC m=+2103.122339523"
Jul 28 22:45:01.015302 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:01.015259    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" podStartSLOduration=1.015242809 podStartE2EDuration="1.015242809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:45:01.013452148 +0000 UTC m=+2103.135800061" watchObservedRunningTime="2026-07-28 22:45:01.015242809 +0000 UTC m=+2103.137590744"
Jul 28 22:45:01.993892 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:01.993856    2560 generic.go:356] "Generic (PLEG): container finished" podID="2ac4035b-a9a7-413e-8fd2-20027d12f074" containerID="de54fb45ce7112ca9066596d5afaa46b1f7316860e589fc92cb434a083f47745" exitCode=0
Jul 28 22:45:01.994403 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:01.993946    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h" event={"ID":"2ac4035b-a9a7-413e-8fd2-20027d12f074","Type":"ContainerDied","Data":"de54fb45ce7112ca9066596d5afaa46b1f7316860e589fc92cb434a083f47745"}
Jul 28 22:45:01.995449 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:01.995429    2560 generic.go:356] "Generic (PLEG): container finished" podID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerID="51fa1f30eacccd51fe9c39342c3987aff8aeed256a8feac5d9e5f39931cc6043" exitCode=7
Jul 28 22:45:01.995541 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:01.995490    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" event={"ID":"bfdae04d-6b5a-4f08-856e-0180f1ac1ee3","Type":"ContainerDied","Data":"51fa1f30eacccd51fe9c39342c3987aff8aeed256a8feac5d9e5f39931cc6043"}
Jul 28 22:45:01.995655 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:01.995641    2560 scope.go:117] "RemoveContainer" containerID="51fa1f30eacccd51fe9c39342c3987aff8aeed256a8feac5d9e5f39931cc6043"
Jul 28 22:45:03.003567 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:03.003528    2560 generic.go:356] "Generic (PLEG): container finished" podID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerID="f472c1bd7d10415375a26451d170bd978b9296fac3c113a785d4d8c1e3f12bea" exitCode=7
Jul 28 22:45:03.004034 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:03.003611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" event={"ID":"bfdae04d-6b5a-4f08-856e-0180f1ac1ee3","Type":"ContainerDied","Data":"f472c1bd7d10415375a26451d170bd978b9296fac3c113a785d4d8c1e3f12bea"}
Jul 28 22:45:03.004034 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:03.003712    2560 scope.go:117] "RemoveContainer" containerID="51fa1f30eacccd51fe9c39342c3987aff8aeed256a8feac5d9e5f39931cc6043"
Jul 28 22:45:03.004034 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:03.003890    2560 scope.go:117] "RemoveContainer" containerID="f472c1bd7d10415375a26451d170bd978b9296fac3c113a785d4d8c1e3f12bea"
Jul 28 22:45:03.004612 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:45:03.004166    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4_odh-ai-gateway-infra(bfdae04d-6b5a-4f08-856e-0180f1ac1ee3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3"
Jul 28 22:45:03.149572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:03.149544    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h"
Jul 28 22:45:04.010482 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:04.010457    2560 scope.go:117] "RemoveContainer" containerID="f472c1bd7d10415375a26451d170bd978b9296fac3c113a785d4d8c1e3f12bea"
Jul 28 22:45:04.010959 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:45:04.010720    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4_odh-ai-gateway-infra(bfdae04d-6b5a-4f08-856e-0180f1ac1ee3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3"
Jul 28 22:45:04.011714 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:04.011689    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h" event={"ID":"2ac4035b-a9a7-413e-8fd2-20027d12f074","Type":"ContainerDied","Data":"40e3282479a6547b7ce2e9f115aaa292093a6224ebfec765e9fdf44a62ac1396"}
Jul 28 22:45:04.011809 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:04.011717    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754645-mfk7h"
Jul 28 22:45:04.011809 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:04.011736    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40e3282479a6547b7ce2e9f115aaa292093a6224ebfec765e9fdf44a62ac1396"
Jul 28 22:45:07.971295 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:07.971261    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:45:17.401658 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:17.401619    2560 scope.go:117] "RemoveContainer" containerID="f472c1bd7d10415375a26451d170bd978b9296fac3c113a785d4d8c1e3f12bea"
Jul 28 22:45:18.091020 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:18.090985    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" event={"ID":"bfdae04d-6b5a-4f08-856e-0180f1ac1ee3","Type":"ContainerStarted","Data":"76822fdcc04a087bd15ba5c870a36c024f8510a2d8393319a3123cad9a3318e2"}
Jul 28 22:45:19.098228 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:19.098197    2560 generic.go:356] "Generic (PLEG): container finished" podID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerID="76822fdcc04a087bd15ba5c870a36c024f8510a2d8393319a3123cad9a3318e2" exitCode=0
Jul 28 22:45:19.098634 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:19.098262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" event={"ID":"bfdae04d-6b5a-4f08-856e-0180f1ac1ee3","Type":"ContainerDied","Data":"76822fdcc04a087bd15ba5c870a36c024f8510a2d8393319a3123cad9a3318e2"}
Jul 28 22:45:19.098634 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:19.098327    2560 scope.go:117] "RemoveContainer" containerID="f472c1bd7d10415375a26451d170bd978b9296fac3c113a785d4d8c1e3f12bea"
Jul 28 22:45:19.114997 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:19.114974    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"]
Jul 28 22:45:20.242330 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:20.242307    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"
Jul 28 22:45:20.401385 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:20.401299    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:20.775157 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:20.775091    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:21.102718 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:21.102619    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:21.112937 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:21.112895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4" event={"ID":"bfdae04d-6b5a-4f08-856e-0180f1ac1ee3","Type":"ContainerDied","Data":"d2f65b34d17a5a302f0034eedbb10e8168823b614397e855e022b2479298178c"}
Jul 28 22:45:21.112937 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:21.112925    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"
Jul 28 22:45:21.113155 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:21.112952    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2f65b34d17a5a302f0034eedbb10e8168823b614397e855e022b2479298178c"
Jul 28 22:45:21.145705 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:21.145672    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"]
Jul 28 22:45:21.151852 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:21.151815    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-104d62-29754645-g96w4"]
Jul 28 22:45:21.215152 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:21.215093    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:22.406934 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:22.406891    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" path="/var/lib/kubelet/pods/bfdae04d-6b5a-4f08-856e-0180f1ac1ee3/volumes"
Jul 28 22:45:24.483376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:24.483337    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:24.581480 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:24.581426    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:24.731343 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:24.731309    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:25.064305 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:25.064267    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:25.363744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:25.363651    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:25.452813 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:25.452777    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:33.441306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:33.441266    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:33.542687 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:33.542631    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:47.808048 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:47.808014    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:47.908326 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:47.908292    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:49.500659 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:49.500625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:52.367002 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.366968    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"]
Jul 28 22:45:52.367572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.367549    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerName="cleanup"
Jul 28 22:45:52.367688 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.367576    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerName="cleanup"
Jul 28 22:45:52.367688 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.367592    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac4035b-a9a7-413e-8fd2-20027d12f074" containerName="cleanup"
Jul 28 22:45:52.367688 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.367601    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac4035b-a9a7-413e-8fd2-20027d12f074" containerName="cleanup"
Jul 28 22:45:52.367688 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.367614    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerName="cleanup"
Jul 28 22:45:52.367688 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.367624    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerName="cleanup"
Jul 28 22:45:52.367949 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.367734    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerName="cleanup"
Jul 28 22:45:52.367949 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.367745    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerName="cleanup"
Jul 28 22:45:52.367949 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.367757    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac4035b-a9a7-413e-8fd2-20027d12f074" containerName="cleanup"
Jul 28 22:45:52.371064 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.371043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.373699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.373672    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-104d62\"/\"openshift-service-ca.crt\""
Jul 28 22:45:52.374968 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.374944    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-104d62\"/\"test-model-104d62-kserve-self-signed-certs\""
Jul 28 22:45:52.375215 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.375121    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-104d62\"/\"kube-root-ca.crt\""
Jul 28 22:45:52.375215 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.374944    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-104d62\"/\"default-dockercfg-hpxm5\""
Jul 28 22:45:52.380638 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.380612    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"]
Jul 28 22:45:52.400222 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.400181    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:45:52.466899 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.466865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276418f-5c36-42bb-829d-32f0b762a79b-tls-certs\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.467089 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.466922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/f276418f-5c36-42bb-829d-32f0b762a79b-kube-api-access-lp2tr\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.467089 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.466968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-home\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.467089 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.466990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-dshm\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.467089 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.467077    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-model-cache\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.467347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.467186    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-tmp-dir\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.568573 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.568541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276418f-5c36-42bb-829d-32f0b762a79b-tls-certs\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.568744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.568589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/f276418f-5c36-42bb-829d-32f0b762a79b-kube-api-access-lp2tr\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.568744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.568623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-home\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.568744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.568642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-dshm\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.568744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.568665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-model-cache\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.568984 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.568789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-tmp-dir\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.569140 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.569081    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-home\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.569257 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.569121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-tmp-dir\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.569257 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.569182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-model-cache\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.570826 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.570808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-dshm\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.570893 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.570874    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276418f-5c36-42bb-829d-32f0b762a79b-tls-certs\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.584480 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.584458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/f276418f-5c36-42bb-829d-32f0b762a79b-kube-api-access-lp2tr\") pod \"test-model-104d62-kserve-c57bfb587-7zrrv\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") " pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.681553 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.681457    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:45:52.829525 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:52.819321    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"]
Jul 28 22:45:52.829525 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:45:52.821771    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf276418f_5c36_42bb_829d_32f0b762a79b.slice/crio-25001178b1c581bc62ef3a286151133e6af93112d75f101e4f5c6951b265c36c WatchSource:0}: Error finding container 25001178b1c581bc62ef3a286151133e6af93112d75f101e4f5c6951b265c36c: Status 404 returned error can't find the container with id 25001178b1c581bc62ef3a286151133e6af93112d75f101e4f5c6951b265c36c
Jul 28 22:45:53.281199 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:53.281160    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv" event={"ID":"f276418f-5c36-42bb-829d-32f0b762a79b","Type":"ContainerStarted","Data":"e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd"}
Jul 28 22:45:53.281416 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:53.281208    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv" event={"ID":"f276418f-5c36-42bb-829d-32f0b762a79b","Type":"ContainerStarted","Data":"25001178b1c581bc62ef3a286151133e6af93112d75f101e4f5c6951b265c36c"}
Jul 28 22:45:53.298774 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:45:53.298726    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv" podStartSLOduration=1.298711315 podStartE2EDuration="1.298711315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:45:53.296294841 +0000 UTC m=+2155.418642735" watchObservedRunningTime="2026-07-28 22:45:53.298711315 +0000 UTC m=+2155.421059222"
Jul 28 22:46:02.682589 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:02.682548    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:46:02.683009 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:02.682603    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:46:02.695457 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:02.695434    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:46:03.343507 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:03.343478    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:46:18.367881 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.367842    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"]
Jul 28 22:46:18.368472 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.368451    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerName="cleanup"
Jul 28 22:46:18.368557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.368476    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerName="cleanup"
Jul 28 22:46:18.368621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.368609    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdae04d-6b5a-4f08-856e-0180f1ac1ee3" containerName="cleanup"
Jul 28 22:46:18.371570 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.371549    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.375430 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.375405    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-72fe47\"/\"test-model-72fe47-kserve-self-signed-certs\""
Jul 28 22:46:18.375430 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.375421    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-72fe47\"/\"openshift-service-ca.crt\""
Jul 28 22:46:18.375683 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.375668    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-72fe47\"/\"default-dockercfg-st9qs\""
Jul 28 22:46:18.375732 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.375680    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-72fe47\"/\"kube-root-ca.crt\""
Jul 28 22:46:18.381391 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.381368    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"]
Jul 28 22:46:18.391733 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.391709    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:46:18.421247 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.421223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tls-certs\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.421388 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.421262    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tmp-dir\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.421388 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.421320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-home\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.421388 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.421338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-model-cache\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.421388 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.421367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-dshm\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.421537 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.421393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzl9\" (UniqueName: \"kubernetes.io/projected/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-kube-api-access-dhzl9\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.522539 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.522499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tls-certs\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.522770 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.522563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tmp-dir\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.522770 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.522626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-home\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.522770 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.522660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-model-cache\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.522770 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.522707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-dshm\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.522770 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.522749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzl9\" (UniqueName: \"kubernetes.io/projected/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-kube-api-access-dhzl9\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.523081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.523061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tmp-dir\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.523205 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.523118    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-model-cache\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.523205 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.523166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-home\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.524963 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.524939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-dshm\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.525079 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.525051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tls-certs\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.530909 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.530886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzl9\" (UniqueName: \"kubernetes.io/projected/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-kube-api-access-dhzl9\") pod \"test-model-72fe47-kserve-5bcc575f5f-ftghd\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") " pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.682776 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.682675    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:18.823140 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:18.823095    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"]
Jul 28 22:46:18.825677 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:46:18.825649    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d973b1a_7e5d_463b_b6a6_b6e8ddce12b0.slice/crio-a740ac21df6ba0ae8dcc63b60939294f47b1edd459ede4f3731f0285687138f4 WatchSource:0}: Error finding container a740ac21df6ba0ae8dcc63b60939294f47b1edd459ede4f3731f0285687138f4: Status 404 returned error can't find the container with id a740ac21df6ba0ae8dcc63b60939294f47b1edd459ede4f3731f0285687138f4
Jul 28 22:46:19.424207 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:19.424170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd" event={"ID":"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0","Type":"ContainerStarted","Data":"cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56"}
Jul 28 22:46:19.424207 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:19.424210    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd" event={"ID":"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0","Type":"ContainerStarted","Data":"a740ac21df6ba0ae8dcc63b60939294f47b1edd459ede4f3731f0285687138f4"}
Jul 28 22:46:19.441728 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:19.441678    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd" podStartSLOduration=1.441663522 podStartE2EDuration="1.441663522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:46:19.439711151 +0000 UTC m=+2181.562059067" watchObservedRunningTime="2026-07-28 22:46:19.441663522 +0000 UTC m=+2181.564011445"
Jul 28 22:46:28.683799 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:28.683756    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:28.684291 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:28.683813    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:28.696873 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:28.696845    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:29.486540 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:29.486512    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:46:54.403357 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:54.403318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:46:59.108153 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:59.108117    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"]
Jul 28 22:46:59.120240 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:46:59.120209    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:00.044305 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:00.044274    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"]
Jul 28 22:47:00.057265 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:00.057237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:10.653681 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.653632    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:10.681921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.681890    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"]
Jul 28 22:47:10.682213 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.682187    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl" podUID="c3013e3a-138d-4966-97e1-4a875b1d4c10" containerName="payload-processing" containerID="cri-o://b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f" gracePeriod=30
Jul 28 22:47:10.764384 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.764345    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:10.899129 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.899078    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:10.939440 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.939414    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:47:10.964069 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.964038    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkb8z\" (UniqueName: \"kubernetes.io/projected/c3013e3a-138d-4966-97e1-4a875b1d4c10-kube-api-access-rkb8z\") pod \"c3013e3a-138d-4966-97e1-4a875b1d4c10\" (UID: \"c3013e3a-138d-4966-97e1-4a875b1d4c10\") "
Jul 28 22:47:10.964242 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.964090    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3013e3a-138d-4966-97e1-4a875b1d4c10-plugins-config-volume\") pod \"c3013e3a-138d-4966-97e1-4a875b1d4c10\" (UID: \"c3013e3a-138d-4966-97e1-4a875b1d4c10\") "
Jul 28 22:47:10.964478 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.964458    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3013e3a-138d-4966-97e1-4a875b1d4c10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3013e3a-138d-4966-97e1-4a875b1d4c10" (UID: "c3013e3a-138d-4966-97e1-4a875b1d4c10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:47:10.966200 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:10.966180    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3013e3a-138d-4966-97e1-4a875b1d4c10-kube-api-access-rkb8z" (OuterVolumeSpecName: "kube-api-access-rkb8z") pod "c3013e3a-138d-4966-97e1-4a875b1d4c10" (UID: "c3013e3a-138d-4966-97e1-4a875b1d4c10"). InnerVolumeSpecName "kube-api-access-rkb8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:47:11.033324 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.033291    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:11.067057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.067012    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkb8z\" (UniqueName: \"kubernetes.io/projected/c3013e3a-138d-4966-97e1-4a875b1d4c10-kube-api-access-rkb8z\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:11.067057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.067053    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3013e3a-138d-4966-97e1-4a875b1d4c10-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:11.308319 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.308281    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:11.701297 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.701201    2560 generic.go:356] "Generic (PLEG): container finished" podID="c3013e3a-138d-4966-97e1-4a875b1d4c10" containerID="b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f" exitCode=0
Jul 28 22:47:11.701297 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.701241    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"
Jul 28 22:47:11.701297 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.701277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl" event={"ID":"c3013e3a-138d-4966-97e1-4a875b1d4c10","Type":"ContainerDied","Data":"b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f"}
Jul 28 22:47:11.701790 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.701334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl" event={"ID":"c3013e3a-138d-4966-97e1-4a875b1d4c10","Type":"ContainerDied","Data":"02b52ef274380305c5a99038f708a5083597869a0712c431e75c5bb554e98e4b"}
Jul 28 22:47:11.701790 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.701363    2560 scope.go:117] "RemoveContainer" containerID="b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f"
Jul 28 22:47:11.713394 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.713368    2560 scope.go:117] "RemoveContainer" containerID="b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f"
Jul 28 22:47:11.713677 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:47:11.713653    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f\": container with ID starting with b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f not found: ID does not exist" containerID="b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f"
Jul 28 22:47:11.713730 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.713687    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f"} err="failed to get container status \"b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f\": rpc error: code = NotFound desc = could not find container \"b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f\": container with ID starting with b5928b864633063a699a48aefa7106a330ff62cfaa7169f62d7610ee7d4b3b4f not found: ID does not exist"
Jul 28 22:47:11.723804 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.723775    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"]
Jul 28 22:47:11.725982 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:11.725961    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-104d62-866ddf5c76-cz9kl"]
Jul 28 22:47:12.407958 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:12.407924    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3013e3a-138d-4966-97e1-4a875b1d4c10" path="/var/lib/kubelet/pods/c3013e3a-138d-4966-97e1-4a875b1d4c10/volumes"
Jul 28 22:47:14.127800 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.127760    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv" podUID="f276418f-5c36-42bb-829d-32f0b762a79b" containerName="main" containerID="cri-o://e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd" gracePeriod=45
Jul 28 22:47:14.373232 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.373205    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:47:14.398382 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-model-cache\") pod \"f276418f-5c36-42bb-829d-32f0b762a79b\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") "
Jul 28 22:47:14.398382 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398349    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/f276418f-5c36-42bb-829d-32f0b762a79b-kube-api-access-lp2tr\") pod \"f276418f-5c36-42bb-829d-32f0b762a79b\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") "
Jul 28 22:47:14.398382 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398366    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-dshm\") pod \"f276418f-5c36-42bb-829d-32f0b762a79b\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") "
Jul 28 22:47:14.398642 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398399    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-home\") pod \"f276418f-5c36-42bb-829d-32f0b762a79b\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") "
Jul 28 22:47:14.398642 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398441    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276418f-5c36-42bb-829d-32f0b762a79b-tls-certs\") pod \"f276418f-5c36-42bb-829d-32f0b762a79b\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") "
Jul 28 22:47:14.398642 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398514    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-tmp-dir\") pod \"f276418f-5c36-42bb-829d-32f0b762a79b\" (UID: \"f276418f-5c36-42bb-829d-32f0b762a79b\") "
Jul 28 22:47:14.398821 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398636    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f276418f-5c36-42bb-829d-32f0b762a79b" (UID: "f276418f-5c36-42bb-829d-32f0b762a79b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:47:14.398821 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398689    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-home" (OuterVolumeSpecName: "home") pod "f276418f-5c36-42bb-829d-32f0b762a79b" (UID: "f276418f-5c36-42bb-829d-32f0b762a79b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:47:14.398921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398850    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f276418f-5c36-42bb-829d-32f0b762a79b" (UID: "f276418f-5c36-42bb-829d-32f0b762a79b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:47:14.398921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398897    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:14.398921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.398917    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:14.400724 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.400702    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-dshm" (OuterVolumeSpecName: "dshm") pod "f276418f-5c36-42bb-829d-32f0b762a79b" (UID: "f276418f-5c36-42bb-829d-32f0b762a79b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:47:14.400724 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.400713    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f276418f-5c36-42bb-829d-32f0b762a79b-kube-api-access-lp2tr" (OuterVolumeSpecName: "kube-api-access-lp2tr") pod "f276418f-5c36-42bb-829d-32f0b762a79b" (UID: "f276418f-5c36-42bb-829d-32f0b762a79b"). InnerVolumeSpecName "kube-api-access-lp2tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:47:14.400886 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.400738    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f276418f-5c36-42bb-829d-32f0b762a79b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f276418f-5c36-42bb-829d-32f0b762a79b" (UID: "f276418f-5c36-42bb-829d-32f0b762a79b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:47:14.499611 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.499574    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276418f-5c36-42bb-829d-32f0b762a79b-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:14.499611 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.499604    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:14.499611 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.499619    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/f276418f-5c36-42bb-829d-32f0b762a79b-kube-api-access-lp2tr\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:14.499850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.499628    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276418f-5c36-42bb-829d-32f0b762a79b-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:14.727605 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.727566    2560 generic.go:356] "Generic (PLEG): container finished" podID="f276418f-5c36-42bb-829d-32f0b762a79b" containerID="e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd" exitCode=0
Jul 28 22:47:14.727778 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.727620    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"
Jul 28 22:47:14.727778 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.727651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv" event={"ID":"f276418f-5c36-42bb-829d-32f0b762a79b","Type":"ContainerDied","Data":"e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd"}
Jul 28 22:47:14.727778 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.727695    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv" event={"ID":"f276418f-5c36-42bb-829d-32f0b762a79b","Type":"ContainerDied","Data":"25001178b1c581bc62ef3a286151133e6af93112d75f101e4f5c6951b265c36c"}
Jul 28 22:47:14.727778 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.727716    2560 scope.go:117] "RemoveContainer" containerID="e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd"
Jul 28 22:47:14.736536 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.736519    2560 scope.go:117] "RemoveContainer" containerID="e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd"
Jul 28 22:47:14.736876 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:47:14.736857    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd\": container with ID starting with e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd not found: ID does not exist" containerID="e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd"
Jul 28 22:47:14.736921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.736886    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd"} err="failed to get container status \"e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd\": rpc error: code = NotFound desc = could not find container \"e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd\": container with ID starting with e97432c775526376b9d3e3f6dd3394b71e14a7b9851d796755730545017225cd not found: ID does not exist"
Jul 28 22:47:14.745132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.745089    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"]
Jul 28 22:47:14.751305 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:14.751278    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-104d62/test-model-104d62-kserve-c57bfb587-7zrrv"]
Jul 28 22:47:15.064068 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.063960    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd" podUID="6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" containerName="main" containerID="cri-o://cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56" gracePeriod=45
Jul 28 22:47:15.313772 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.313741    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:47:15.406689 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.406600    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-home\") pod \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") "
Jul 28 22:47:15.406689 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.406655    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tmp-dir\") pod \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") "
Jul 28 22:47:15.406913 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.406701    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-model-cache\") pod \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") "
Jul 28 22:47:15.406913 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.406720    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-dshm\") pod \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") "
Jul 28 22:47:15.406913 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.406792    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tls-certs\") pod \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") "
Jul 28 22:47:15.406913 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.406819    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhzl9\" (UniqueName: \"kubernetes.io/projected/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-kube-api-access-dhzl9\") pod \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\" (UID: \"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0\") "
Jul 28 22:47:15.407165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.406968    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-home" (OuterVolumeSpecName: "home") pod "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" (UID: "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:47:15.407165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.406981    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" (UID: "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:47:15.407165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.406974    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" (UID: "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:47:15.407165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.407123    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:15.407165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.407137    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:15.407165 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.407147    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:15.409057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.409021    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-kube-api-access-dhzl9" (OuterVolumeSpecName: "kube-api-access-dhzl9") pod "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" (UID: "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0"). InnerVolumeSpecName "kube-api-access-dhzl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:47:15.409057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.409026    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-dshm" (OuterVolumeSpecName: "dshm") pod "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" (UID: "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:47:15.409057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.409041    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" (UID: "6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:47:15.507755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.507712    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:15.507755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.507750    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhzl9\" (UniqueName: \"kubernetes.io/projected/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-kube-api-access-dhzl9\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:15.507755 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.507763    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:15.735080 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.735042    2560 generic.go:356] "Generic (PLEG): container finished" podID="6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" containerID="cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56" exitCode=0
Jul 28 22:47:15.735278 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.735084    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"
Jul 28 22:47:15.735278 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.735123    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd" event={"ID":"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0","Type":"ContainerDied","Data":"cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56"}
Jul 28 22:47:15.735278 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.735163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd" event={"ID":"6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0","Type":"ContainerDied","Data":"a740ac21df6ba0ae8dcc63b60939294f47b1edd459ede4f3731f0285687138f4"}
Jul 28 22:47:15.735278 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.735180    2560 scope.go:117] "RemoveContainer" containerID="cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56"
Jul 28 22:47:15.744535 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.744515    2560 scope.go:117] "RemoveContainer" containerID="cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56"
Jul 28 22:47:15.744849 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:47:15.744814    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56\": container with ID starting with cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56 not found: ID does not exist" containerID="cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56"
Jul 28 22:47:15.744962 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.744858    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56"} err="failed to get container status \"cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56\": rpc error: code = NotFound desc = could not find container \"cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56\": container with ID starting with cad6a0e12d83ee23a92e83855794ff69340bfe20a60e6c8a9f0bc052a535ef56 not found: ID does not exist"
Jul 28 22:47:15.755771 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.755743    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"]
Jul 28 22:47:15.759329 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:15.759308    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-72fe47/test-model-72fe47-kserve-5bcc575f5f-ftghd"]
Jul 28 22:47:16.407742 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:16.407700    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" path="/var/lib/kubelet/pods/6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0/volumes"
Jul 28 22:47:16.408317 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:16.408296    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f276418f-5c36-42bb-829d-32f0b762a79b" path="/var/lib/kubelet/pods/f276418f-5c36-42bb-829d-32f0b762a79b/volumes"
Jul 28 22:47:17.136715 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.136676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:17.237612 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.237564    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:17.373965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.373933    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:17.613741 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.613709    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:17.722854 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.722815    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"]
Jul 28 22:47:17.723307 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.723289    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" containerName="main"
Jul 28 22:47:17.723307 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.723308    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" containerName="main"
Jul 28 22:47:17.723471 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.723326    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f276418f-5c36-42bb-829d-32f0b762a79b" containerName="main"
Jul 28 22:47:17.723471 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.723334    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f276418f-5c36-42bb-829d-32f0b762a79b" containerName="main"
Jul 28 22:47:17.723471 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.723358    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3013e3a-138d-4966-97e1-4a875b1d4c10" containerName="payload-processing"
Jul 28 22:47:17.723471 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.723366    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3013e3a-138d-4966-97e1-4a875b1d4c10" containerName="payload-processing"
Jul 28 22:47:17.723471 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.723468    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f276418f-5c36-42bb-829d-32f0b762a79b" containerName="main"
Jul 28 22:47:17.723711 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.723485    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3013e3a-138d-4966-97e1-4a875b1d4c10" containerName="payload-processing"
Jul 28 22:47:17.723711 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.723495    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d973b1a-7e5d-463b-b6a6-b6e8ddce12b0" containerName="main"
Jul 28 22:47:17.727701 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.727678    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"
Jul 28 22:47:17.730306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.730075    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 22:47:17.733221 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.733198    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"]
Jul 28 22:47:17.829877 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.829843    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7\" (UID: \"62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"
Jul 28 22:47:17.931301 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.931199    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7\" (UID: \"62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"
Jul 28 22:47:17.931813 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:17.931792    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7\" (UID: \"62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"
Jul 28 22:47:18.039472 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:18.039434    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"
Jul 28 22:47:18.175448 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:18.175424    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"]
Jul 28 22:47:18.177742 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:47:18.177706    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e74ecc_13fb_4b8c_bcdb_56dd9b1370aa.slice/crio-fd5d202362981913dc68ef647f8c2a4747560755eff1c419c8c54d6576aebf4b WatchSource:0}: Error finding container fd5d202362981913dc68ef647f8c2a4747560755eff1c419c8c54d6576aebf4b: Status 404 returned error can't find the container with id fd5d202362981913dc68ef647f8c2a4747560755eff1c419c8c54d6576aebf4b
Jul 28 22:47:18.754093 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:18.753990    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7" event={"ID":"62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa","Type":"ContainerStarted","Data":"c08cb4409c2927ea9ea7b9c30447ad8a60818aafdf794ed51b7a59c70cfb3122"}
Jul 28 22:47:18.754093 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:18.754058    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7" event={"ID":"62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa","Type":"ContainerStarted","Data":"fd5d202362981913dc68ef647f8c2a4747560755eff1c419c8c54d6576aebf4b"}
Jul 28 22:47:18.775142 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:18.775065    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7" podStartSLOduration=1.775048956 podStartE2EDuration="1.775048956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:47:18.773062815 +0000 UTC m=+2240.895410729" watchObservedRunningTime="2026-07-28 22:47:18.775048956 +0000 UTC m=+2240.897396848"
Jul 28 22:47:19.764680 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:19.764640    2560 generic.go:356] "Generic (PLEG): container finished" podID="62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa" containerID="c08cb4409c2927ea9ea7b9c30447ad8a60818aafdf794ed51b7a59c70cfb3122" exitCode=0
Jul 28 22:47:19.765090 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:19.764726    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7" event={"ID":"62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa","Type":"ContainerDied","Data":"c08cb4409c2927ea9ea7b9c30447ad8a60818aafdf794ed51b7a59c70cfb3122"}
Jul 28 22:47:20.902055 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:20.902029    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"
Jul 28 22:47:20.964329 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:20.964292    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa-maas-api-service-ca\") pod \"62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa\" (UID: \"62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa\") "
Jul 28 22:47:20.964727 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:20.964705    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa" (UID: "62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:47:21.065877 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:21.065787    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:47:21.777723 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:21.777696    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"
Jul 28 22:47:21.777921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:21.777697    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7" event={"ID":"62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa","Type":"ContainerDied","Data":"fd5d202362981913dc68ef647f8c2a4747560755eff1c419c8c54d6576aebf4b"}
Jul 28 22:47:21.777921 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:21.777826    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd5d202362981913dc68ef647f8c2a4747560755eff1c419c8c54d6576aebf4b"
Jul 28 22:47:28.130210 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:28.130164    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:28.198904 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:28.198865    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:28.257515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:28.257479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:28.306362 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:28.306288    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"]
Jul 28 22:47:28.308960 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:28.308933    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72fe47-rbxl7"]
Jul 28 22:47:28.314826 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:28.314781    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:28.407749 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:28.407672    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa" path="/var/lib/kubelet/pods/62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa/volumes"
Jul 28 22:47:28.694326 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:28.694232    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:34.608770 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:34.608732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:34.709565 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:34.709528    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:34.891672 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:34.891560    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:35.101320 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:35.101280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:35.437706 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:35.437668    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:35.756243 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:35.756210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:35.857044 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:35.857004    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:38.234633 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.234594    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:38.333487 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.333448    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:38.375398 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.375365    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"]
Jul 28 22:47:38.376042 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.376019    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa" containerName="revoke-keys"
Jul 28 22:47:38.376130 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.376047    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa" containerName="revoke-keys"
Jul 28 22:47:38.376199 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.376167    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e74ecc-13fb-4b8c-bcdb-56dd9b1370aa" containerName="revoke-keys"
Jul 28 22:47:38.379772 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.379749    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:38.382602 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.382483    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-63f77a-serving-cert\""
Jul 28 22:47:38.382754 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.382717    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-89lsd\""
Jul 28 22:47:38.390204 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.390170    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"]
Jul 28 22:47:38.411532 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.411501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:38.535152 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.535047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a564fb6-ae5c-4d12-967a-2a3b938273db-maas-api-tls\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:38.535482 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.535455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a564fb6-ae5c-4d12-967a-2a3b938273db-openshift-service-ca\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:38.535611 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.535533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chnd\" (UniqueName: \"kubernetes.io/projected/7a564fb6-ae5c-4d12-967a-2a3b938273db-kube-api-access-8chnd\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:38.636188 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.636152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a564fb6-ae5c-4d12-967a-2a3b938273db-openshift-service-ca\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:38.636373 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.636198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chnd\" (UniqueName: \"kubernetes.io/projected/7a564fb6-ae5c-4d12-967a-2a3b938273db-kube-api-access-8chnd\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:38.636373 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.636322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a564fb6-ae5c-4d12-967a-2a3b938273db-maas-api-tls\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:38.636508 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:47:38.636448    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-serving-cert: secret "maas-api-e2e-api-a-63f77a-serving-cert" not found
Jul 28 22:47:38.636508 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:47:38.636505    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a564fb6-ae5c-4d12-967a-2a3b938273db-maas-api-tls podName:7a564fb6-ae5c-4d12-967a-2a3b938273db nodeName:}" failed. No retries permitted until 2026-07-28 22:47:39.136489627 +0000 UTC m=+2261.258837503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a564fb6-ae5c-4d12-967a-2a3b938273db-maas-api-tls") pod "maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp" (UID: "7a564fb6-ae5c-4d12-967a-2a3b938273db") : secret "maas-api-e2e-api-a-63f77a-serving-cert" not found
Jul 28 22:47:38.636860 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.636837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a564fb6-ae5c-4d12-967a-2a3b938273db-openshift-service-ca\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:38.645795 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.645776    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chnd\" (UniqueName: \"kubernetes.io/projected/7a564fb6-ae5c-4d12-967a-2a3b938273db-kube-api-access-8chnd\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:38.843086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:38.842754    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:39.010839 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:39.010794    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:39.116933 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:39.116847    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:47:39.141020 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:39.140990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a564fb6-ae5c-4d12-967a-2a3b938273db-maas-api-tls\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:39.143702 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:39.143680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a564fb6-ae5c-4d12-967a-2a3b938273db-maas-api-tls\") pod \"maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:39.292275 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:39.292239    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:39.424992 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:39.424968    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"]
Jul 28 22:47:39.427214 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:47:39.427178    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a564fb6_ae5c_4d12_967a_2a3b938273db.slice/crio-cfb221fa592a33ae2194897c1e549e6defcde2ede0ec64718f44981d07dcefb3 WatchSource:0}: Error finding container cfb221fa592a33ae2194897c1e549e6defcde2ede0ec64718f44981d07dcefb3: Status 404 returned error can't find the container with id cfb221fa592a33ae2194897c1e549e6defcde2ede0ec64718f44981d07dcefb3
Jul 28 22:47:39.428525 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:39.428509    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:47:39.874642 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:39.874605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp" event={"ID":"7a564fb6-ae5c-4d12-967a-2a3b938273db","Type":"ContainerStarted","Data":"cfb221fa592a33ae2194897c1e549e6defcde2ede0ec64718f44981d07dcefb3"}
Jul 28 22:47:40.880677 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:40.880637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp" event={"ID":"7a564fb6-ae5c-4d12-967a-2a3b938273db","Type":"ContainerStarted","Data":"104f6400164d101be6e2012cb3bb920e6d05b8e74602d716fd13fd0ba5d05fe3"}
Jul 28 22:47:40.881094 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:40.880704    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:40.899249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:40.899200    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp" podStartSLOduration=2.304263146 podStartE2EDuration="2.8991793s" totalImagesPullingTime="594.916154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:47:40.896815926 +0000 UTC m=+2263.019163818" watchObservedRunningTime="2026-07-28 22:47:40.8991793 +0000 UTC m=+2263.021527278"
Jul 28 22:47:46.892376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:46.892345    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:47:58.622676 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:47:58.622644    2560 scope.go:117] "RemoveContainer" containerID="560be0b24946d0f485fab9c2b288c851b69acf7f690ffe9fc9894806dc11dc27"
Jul 28 22:48:03.257934 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:03.257895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:03.795988 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:03.795950    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:04.071054 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:04.070963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:04.226625 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:04.226589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:07.514810 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:07.514777    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:07.588592 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:07.588552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:07.651669 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:07.651632    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:07.849002 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:07.848912    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:08.188418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.188331    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:08.284027 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.283986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:08.285191 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.285168    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"]
Jul 28 22:48:08.288818 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.288802    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:08.291325 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.291306    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-472fa3-dockercfg-rhgmk\""
Jul 28 22:48:08.291454 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.291323    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-472fa3\""
Jul 28 22:48:08.298717 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.298694    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"]
Jul 28 22:48:08.426176 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.426138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pjm5\" (UniqueName: \"kubernetes.io/projected/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-kube-api-access-6pjm5\") pod \"payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s\" (UID: \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:08.426361 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.426196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s\" (UID: \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:08.527699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.527657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pjm5\" (UniqueName: \"kubernetes.io/projected/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-kube-api-access-6pjm5\") pod \"payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s\" (UID: \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:08.528168 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.527707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s\" (UID: \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:08.528554 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.528533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s\" (UID: \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:08.535473 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.535444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pjm5\" (UniqueName: \"kubernetes.io/projected/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-kube-api-access-6pjm5\") pod \"payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s\" (UID: \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:08.599521 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.599485    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:08.973597 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:08.973569    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"]
Jul 28 22:48:08.975918 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:48:08.975888    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089fc9dc_5f4f_4ce2_a5c3_cd74fb4e4b33.slice/crio-c7103d369bb1f10e91eb9ef249327e766a5339b5030e934073f8f263fa328e3f WatchSource:0}: Error finding container c7103d369bb1f10e91eb9ef249327e766a5339b5030e934073f8f263fa328e3f: Status 404 returned error can't find the container with id c7103d369bb1f10e91eb9ef249327e766a5339b5030e934073f8f263fa328e3f
Jul 28 22:48:09.034123 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:09.034070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s" event={"ID":"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33","Type":"ContainerStarted","Data":"c7103d369bb1f10e91eb9ef249327e766a5339b5030e934073f8f263fa328e3f"}
Jul 28 22:48:10.049440 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:10.049393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s" event={"ID":"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33","Type":"ContainerStarted","Data":"8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0"}
Jul 28 22:48:10.049919 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:10.049544    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:10.063859 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:10.063809    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s" podStartSLOduration=2.06379379 podStartE2EDuration="2.06379379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:48:10.063090981 +0000 UTC m=+2292.185438875" watchObservedRunningTime="2026-07-28 22:48:10.06379379 +0000 UTC m=+2292.186141682"
Jul 28 22:48:21.056780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:21.056750    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:48:32.424251 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:32.424208    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:35.377566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:35.377486    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"]
Jul 28 22:48:35.381266 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:35.381240    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"
Jul 28 22:48:35.383635 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:35.383614    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 22:48:35.389996 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:35.389963    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"]
Jul 28 22:48:35.488555 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:35.488516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b1d706-7492-488d-8201-241f9bba9758-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq\" (UID: \"77b1d706-7492-488d-8201-241f9bba9758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"
Jul 28 22:48:35.589873 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:35.589840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b1d706-7492-488d-8201-241f9bba9758-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq\" (UID: \"77b1d706-7492-488d-8201-241f9bba9758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"
Jul 28 22:48:35.590489 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:35.590466    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b1d706-7492-488d-8201-241f9bba9758-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq\" (UID: \"77b1d706-7492-488d-8201-241f9bba9758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"
Jul 28 22:48:35.693327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:35.693232    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"
Jul 28 22:48:35.828690 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:35.828663    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"]
Jul 28 22:48:35.830767 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:48:35.830713    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b1d706_7492_488d_8201_241f9bba9758.slice/crio-9e4f490f9d4dcaaf53038641cf65578cf1660667f1c1fd351ce5c445c1043325 WatchSource:0}: Error finding container 9e4f490f9d4dcaaf53038641cf65578cf1660667f1c1fd351ce5c445c1043325: Status 404 returned error can't find the container with id 9e4f490f9d4dcaaf53038641cf65578cf1660667f1c1fd351ce5c445c1043325
Jul 28 22:48:36.187397 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:36.187361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq" event={"ID":"77b1d706-7492-488d-8201-241f9bba9758","Type":"ContainerStarted","Data":"90988a2b7b251f334fb7fad2ad653f67060197a4ea32ba8f76afac665d908b2a"}
Jul 28 22:48:36.187548 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:36.187417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq" event={"ID":"77b1d706-7492-488d-8201-241f9bba9758","Type":"ContainerStarted","Data":"9e4f490f9d4dcaaf53038641cf65578cf1660667f1c1fd351ce5c445c1043325"}
Jul 28 22:48:36.203294 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:36.203242    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq" podStartSLOduration=1.203226547 podStartE2EDuration="1.203226547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:48:36.202648951 +0000 UTC m=+2318.324996843" watchObservedRunningTime="2026-07-28 22:48:36.203226547 +0000 UTC m=+2318.325574472"
Jul 28 22:48:37.193781 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:37.193745    2560 generic.go:356] "Generic (PLEG): container finished" podID="77b1d706-7492-488d-8201-241f9bba9758" containerID="90988a2b7b251f334fb7fad2ad653f67060197a4ea32ba8f76afac665d908b2a" exitCode=0
Jul 28 22:48:37.194170 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:37.193836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq" event={"ID":"77b1d706-7492-488d-8201-241f9bba9758","Type":"ContainerDied","Data":"90988a2b7b251f334fb7fad2ad653f67060197a4ea32ba8f76afac665d908b2a"}
Jul 28 22:48:38.327189 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:38.327165    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"
Jul 28 22:48:38.417801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:38.417768    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b1d706-7492-488d-8201-241f9bba9758-maas-api-service-ca\") pod \"77b1d706-7492-488d-8201-241f9bba9758\" (UID: \"77b1d706-7492-488d-8201-241f9bba9758\") "
Jul 28 22:48:38.418093 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:38.418072    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b1d706-7492-488d-8201-241f9bba9758-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77b1d706-7492-488d-8201-241f9bba9758" (UID: "77b1d706-7492-488d-8201-241f9bba9758"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:48:38.518834 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:38.518782    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b1d706-7492-488d-8201-241f9bba9758-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:48:39.205161 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:39.205115    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"
Jul 28 22:48:39.205161 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:39.205144    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq" event={"ID":"77b1d706-7492-488d-8201-241f9bba9758","Type":"ContainerDied","Data":"9e4f490f9d4dcaaf53038641cf65578cf1660667f1c1fd351ce5c445c1043325"}
Jul 28 22:48:39.205418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:39.205190    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e4f490f9d4dcaaf53038641cf65578cf1660667f1c1fd351ce5c445c1043325"
Jul 28 22:48:46.034443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.034408    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"]
Jul 28 22:48:46.034863 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.034605    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp" podUID="7a564fb6-ae5c-4d12-967a-2a3b938273db" containerName="maas-api" containerID="cri-o://104f6400164d101be6e2012cb3bb920e6d05b8e74602d716fd13fd0ba5d05fe3" gracePeriod=30
Jul 28 22:48:46.065305 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.065273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:46.166635 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.166596    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:46.246778 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.246749    2560 generic.go:356] "Generic (PLEG): container finished" podID="7a564fb6-ae5c-4d12-967a-2a3b938273db" containerID="104f6400164d101be6e2012cb3bb920e6d05b8e74602d716fd13fd0ba5d05fe3" exitCode=0
Jul 28 22:48:46.246935 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.246793    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp" event={"ID":"7a564fb6-ae5c-4d12-967a-2a3b938273db","Type":"ContainerDied","Data":"104f6400164d101be6e2012cb3bb920e6d05b8e74602d716fd13fd0ba5d05fe3"}
Jul 28 22:48:46.266674 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.266647    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:46.288500 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.288433    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:48:46.355946 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.355862    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"]
Jul 28 22:48:46.357762 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.357741    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-63f77a-7s2jq"]
Jul 28 22:48:46.394943 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.394914    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a564fb6-ae5c-4d12-967a-2a3b938273db-openshift-service-ca\") pod \"7a564fb6-ae5c-4d12-967a-2a3b938273db\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") "
Jul 28 22:48:46.395132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.395010    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a564fb6-ae5c-4d12-967a-2a3b938273db-maas-api-tls\") pod \"7a564fb6-ae5c-4d12-967a-2a3b938273db\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") "
Jul 28 22:48:46.395132 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.395037    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8chnd\" (UniqueName: \"kubernetes.io/projected/7a564fb6-ae5c-4d12-967a-2a3b938273db-kube-api-access-8chnd\") pod \"7a564fb6-ae5c-4d12-967a-2a3b938273db\" (UID: \"7a564fb6-ae5c-4d12-967a-2a3b938273db\") "
Jul 28 22:48:46.395386 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.395326    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a564fb6-ae5c-4d12-967a-2a3b938273db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a564fb6-ae5c-4d12-967a-2a3b938273db" (UID: "7a564fb6-ae5c-4d12-967a-2a3b938273db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:48:46.397326 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.397297    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a564fb6-ae5c-4d12-967a-2a3b938273db-kube-api-access-8chnd" (OuterVolumeSpecName: "kube-api-access-8chnd") pod "7a564fb6-ae5c-4d12-967a-2a3b938273db" (UID: "7a564fb6-ae5c-4d12-967a-2a3b938273db"). InnerVolumeSpecName "kube-api-access-8chnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:48:46.397436 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.397297    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a564fb6-ae5c-4d12-967a-2a3b938273db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a564fb6-ae5c-4d12-967a-2a3b938273db" (UID: "7a564fb6-ae5c-4d12-967a-2a3b938273db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:48:46.408331 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.408306    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77b1d706-7492-488d-8201-241f9bba9758" path="/var/lib/kubelet/pods/77b1d706-7492-488d-8201-241f9bba9758/volumes"
Jul 28 22:48:46.496662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.496625    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a564fb6-ae5c-4d12-967a-2a3b938273db-openshift-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:48:46.496662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.496656    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a564fb6-ae5c-4d12-967a-2a3b938273db-maas-api-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:48:46.496662 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.496666    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8chnd\" (UniqueName: \"kubernetes.io/projected/7a564fb6-ae5c-4d12-967a-2a3b938273db-kube-api-access-8chnd\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:48:46.729711 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:46.729673    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:47.252962 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:47.252929    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"
Jul 28 22:48:47.253461 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:47.252957    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp" event={"ID":"7a564fb6-ae5c-4d12-967a-2a3b938273db","Type":"ContainerDied","Data":"cfb221fa592a33ae2194897c1e549e6defcde2ede0ec64718f44981d07dcefb3"}
Jul 28 22:48:47.253461 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:47.253056    2560 scope.go:117] "RemoveContainer" containerID="104f6400164d101be6e2012cb3bb920e6d05b8e74602d716fd13fd0ba5d05fe3"
Jul 28 22:48:47.270615 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:47.270593    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"]
Jul 28 22:48:47.274654 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:47.274633    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-63f77a-6c4fbff78-mldgp"]
Jul 28 22:48:48.407561 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:48.407527    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a564fb6-ae5c-4d12-967a-2a3b938273db" path="/var/lib/kubelet/pods/7a564fb6-ae5c-4d12-967a-2a3b938273db/volumes"
Jul 28 22:48:52.627301 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:52.627218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:52.727074 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:52.727043    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:52.883073 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:52.882996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:53.113356 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.113325    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:48:53.227833 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.227797    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"]
Jul 28 22:48:53.228256 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.228241    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a564fb6-ae5c-4d12-967a-2a3b938273db" containerName="maas-api"
Jul 28 22:48:53.228303 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.228259    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a564fb6-ae5c-4d12-967a-2a3b938273db" containerName="maas-api"
Jul 28 22:48:53.228303 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.228285    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b1d706-7492-488d-8201-241f9bba9758" containerName="revoke-keys"
Jul 28 22:48:53.228303 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.228291    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b1d706-7492-488d-8201-241f9bba9758" containerName="revoke-keys"
Jul 28 22:48:53.228401 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.228357    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a564fb6-ae5c-4d12-967a-2a3b938273db" containerName="maas-api"
Jul 28 22:48:53.228401 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.228370    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b1d706-7492-488d-8201-241f9bba9758" containerName="revoke-keys"
Jul 28 22:48:53.232758 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.232742    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"
Jul 28 22:48:53.236749 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.236729    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 22:48:53.242284 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.242261    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"]
Jul 28 22:48:53.362980 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.362945    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb1125-7cf6-4788-84e9-0fbb28ff66e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc\" (UID: \"13fb1125-7cf6-4788-84e9-0fbb28ff66e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"
Jul 28 22:48:53.464382 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.464349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb1125-7cf6-4788-84e9-0fbb28ff66e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc\" (UID: \"13fb1125-7cf6-4788-84e9-0fbb28ff66e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"
Jul 28 22:48:53.464977 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.464956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb1125-7cf6-4788-84e9-0fbb28ff66e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc\" (UID: \"13fb1125-7cf6-4788-84e9-0fbb28ff66e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"
Jul 28 22:48:53.544468 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.544381    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"
Jul 28 22:48:53.676561 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:53.676513    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"]
Jul 28 22:48:53.679482 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:48:53.679454    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fb1125_7cf6_4788_84e9_0fbb28ff66e4.slice/crio-1625960448df12f30f62551b2f998c90c7fe1ebf9fad5bed4c0d908725641c7f WatchSource:0}: Error finding container 1625960448df12f30f62551b2f998c90c7fe1ebf9fad5bed4c0d908725641c7f: Status 404 returned error can't find the container with id 1625960448df12f30f62551b2f998c90c7fe1ebf9fad5bed4c0d908725641c7f
Jul 28 22:48:54.293523 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:54.293488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc" event={"ID":"13fb1125-7cf6-4788-84e9-0fbb28ff66e4","Type":"ContainerStarted","Data":"5756e07f007665ecee3b9d38cbc7cf49b03381f146f117cf678d426ccbf1125f"}
Jul 28 22:48:54.293699 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:54.293539    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc" event={"ID":"13fb1125-7cf6-4788-84e9-0fbb28ff66e4","Type":"ContainerStarted","Data":"1625960448df12f30f62551b2f998c90c7fe1ebf9fad5bed4c0d908725641c7f"}
Jul 28 22:48:54.307885 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:54.307836    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc" podStartSLOduration=1.307814983 podStartE2EDuration="1.307814983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:48:54.306865924 +0000 UTC m=+2336.429213819" watchObservedRunningTime="2026-07-28 22:48:54.307814983 +0000 UTC m=+2336.430162877"
Jul 28 22:48:55.299679 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:55.299634    2560 generic.go:356] "Generic (PLEG): container finished" podID="13fb1125-7cf6-4788-84e9-0fbb28ff66e4" containerID="5756e07f007665ecee3b9d38cbc7cf49b03381f146f117cf678d426ccbf1125f" exitCode=0
Jul 28 22:48:55.300181 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:55.299759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc" event={"ID":"13fb1125-7cf6-4788-84e9-0fbb28ff66e4","Type":"ContainerDied","Data":"5756e07f007665ecee3b9d38cbc7cf49b03381f146f117cf678d426ccbf1125f"}
Jul 28 22:48:56.437034 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:56.437001    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"
Jul 28 22:48:56.596451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:56.596362    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb1125-7cf6-4788-84e9-0fbb28ff66e4-maas-api-service-ca\") pod \"13fb1125-7cf6-4788-84e9-0fbb28ff66e4\" (UID: \"13fb1125-7cf6-4788-84e9-0fbb28ff66e4\") "
Jul 28 22:48:56.596715 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:56.596692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fb1125-7cf6-4788-84e9-0fbb28ff66e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13fb1125-7cf6-4788-84e9-0fbb28ff66e4" (UID: "13fb1125-7cf6-4788-84e9-0fbb28ff66e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:48:56.698205 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:56.698172    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb1125-7cf6-4788-84e9-0fbb28ff66e4-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:48:57.311164 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:57.311131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc" event={"ID":"13fb1125-7cf6-4788-84e9-0fbb28ff66e4","Type":"ContainerDied","Data":"1625960448df12f30f62551b2f998c90c7fe1ebf9fad5bed4c0d908725641c7f"}
Jul 28 22:48:57.311341 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:57.311184    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1625960448df12f30f62551b2f998c90c7fe1ebf9fad5bed4c0d908725641c7f"
Jul 28 22:48:57.311341 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:48:57.311145    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"
Jul 28 22:49:04.309525 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.309481    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:04.333584 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.333548    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"]
Jul 28 22:49:04.333873 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.333835    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s" podUID="089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33" containerName="payload-pre-processing" containerID="cri-o://8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0" gracePeriod=30
Jul 28 22:49:04.409230 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.409173    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:04.465896 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.465717    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"]
Jul 28 22:49:04.468127 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.468085    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-472fa3-pt6kc"]
Jul 28 22:49:04.470121 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.470068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:04.595385 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.595360    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:49:04.669427 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.669391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-plugins-config-volume\") pod \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\" (UID: \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\") "
Jul 28 22:49:04.669605 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.669448    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pjm5\" (UniqueName: \"kubernetes.io/projected/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-kube-api-access-6pjm5\") pod \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\" (UID: \"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33\") "
Jul 28 22:49:04.669872 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.669842    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33" (UID: "089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:49:04.671561 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.671543    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-kube-api-access-6pjm5" (OuterVolumeSpecName: "kube-api-access-6pjm5") pod "089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33" (UID: "089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33"). InnerVolumeSpecName "kube-api-access-6pjm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:49:04.771214 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.771180    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:49:04.771214 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.771209    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pjm5\" (UniqueName: \"kubernetes.io/projected/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33-kube-api-access-6pjm5\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:49:04.838554 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:04.838457    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:05.355716 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:05.355678    2560 generic.go:356] "Generic (PLEG): container finished" podID="089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33" containerID="8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0" exitCode=0
Jul 28 22:49:05.356192 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:05.355741    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"
Jul 28 22:49:05.356192 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:05.355760    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s" event={"ID":"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33","Type":"ContainerDied","Data":"8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0"}
Jul 28 22:49:05.356192 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:05.355848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s" event={"ID":"089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33","Type":"ContainerDied","Data":"c7103d369bb1f10e91eb9ef249327e766a5339b5030e934073f8f263fa328e3f"}
Jul 28 22:49:05.356192 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:05.355872    2560 scope.go:117] "RemoveContainer" containerID="8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0"
Jul 28 22:49:05.366140 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:05.366091    2560 scope.go:117] "RemoveContainer" containerID="8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0"
Jul 28 22:49:05.366623 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:49:05.366602    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0\": container with ID starting with 8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0 not found: ID does not exist" containerID="8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0"
Jul 28 22:49:05.366797 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:05.366774    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0"} err="failed to get container status \"8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0\": rpc error: code = NotFound desc = could not find container \"8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0\": container with ID starting with 8c68a3a86770767a191bf8aed1e003c930712d30b1d77e408ed1d3d5617364d0 not found: ID does not exist"
Jul 28 22:49:05.378212 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:05.378183    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"]
Jul 28 22:49:05.382087 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:05.382063    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-472fa3-6479cf8bc6-9mm9s"]
Jul 28 22:49:06.407537 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:06.407505    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33" path="/var/lib/kubelet/pods/089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33/volumes"
Jul 28 22:49:06.407953 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:06.407837    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13fb1125-7cf6-4788-84e9-0fbb28ff66e4" path="/var/lib/kubelet/pods/13fb1125-7cf6-4788-84e9-0fbb28ff66e4/volumes"
Jul 28 22:49:10.572369 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:10.572325    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:10.672330 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:10.672293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:10.839482 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:10.839389    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:11.080069 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:11.080027    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:11.421219 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:11.421185    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:11.717753 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:11.717707    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:11.971479 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:11.971382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:14.810463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:14.810430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:14.890347 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:14.890314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:14.963230 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:14.963197    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:15.584835 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.584804    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:15.682934 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.682899    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:15.756550 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.756516    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"]
Jul 28 22:49:15.756916 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.756904    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33" containerName="payload-pre-processing"
Jul 28 22:49:15.756961 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.756918    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33" containerName="payload-pre-processing"
Jul 28 22:49:15.756961 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.756929    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13fb1125-7cf6-4788-84e9-0fbb28ff66e4" containerName="revoke-keys"
Jul 28 22:49:15.756961 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.756935    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13fb1125-7cf6-4788-84e9-0fbb28ff66e4" containerName="revoke-keys"
Jul 28 22:49:15.757050 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.757022    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="089fc9dc-5f4f-4ce2-a5c3-cd74fb4e4b33" containerName="payload-pre-processing"
Jul 28 22:49:15.757050 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.757033    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="13fb1125-7cf6-4788-84e9-0fbb28ff66e4" containerName="revoke-keys"
Jul 28 22:49:15.761648 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.761629    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:15.764227 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.764204    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-a134df-dockercfg-l8vgj\""
Jul 28 22:49:15.764340 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.764211    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-a134df\""
Jul 28 22:49:15.773965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.773944    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"]
Jul 28 22:49:15.877980 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.877898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ks9x\" (UniqueName: \"kubernetes.io/projected/5bcc4db3-9814-4b43-a4a8-96f0232c052d-kube-api-access-4ks9x\") pod \"payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9\" (UID: \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:15.877980 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.877938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcc4db3-9814-4b43-a4a8-96f0232c052d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9\" (UID: \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:15.978676 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.978640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ks9x\" (UniqueName: \"kubernetes.io/projected/5bcc4db3-9814-4b43-a4a8-96f0232c052d-kube-api-access-4ks9x\") pod \"payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9\" (UID: \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:15.978880 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.978701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcc4db3-9814-4b43-a4a8-96f0232c052d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9\" (UID: \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:15.979404 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.979385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcc4db3-9814-4b43-a4a8-96f0232c052d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9\" (UID: \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:15.986229 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:15.986208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ks9x\" (UniqueName: \"kubernetes.io/projected/5bcc4db3-9814-4b43-a4a8-96f0232c052d-kube-api-access-4ks9x\") pod \"payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9\" (UID: \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:16.071718 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:16.071681    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:16.203966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:16.203940    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"]
Jul 28 22:49:16.205968 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:49:16.205941    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcc4db3_9814_4b43_a4a8_96f0232c052d.slice/crio-8b0310916d8ad90056ef7b60b587fef695656f6855e938437afddffa644c5b03 WatchSource:0}: Error finding container 8b0310916d8ad90056ef7b60b587fef695656f6855e938437afddffa644c5b03: Status 404 returned error can't find the container with id 8b0310916d8ad90056ef7b60b587fef695656f6855e938437afddffa644c5b03
Jul 28 22:49:16.251225 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:16.251195    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:16.418982 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:16.418886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9" event={"ID":"5bcc4db3-9814-4b43-a4a8-96f0232c052d","Type":"ContainerStarted","Data":"36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4"}
Jul 28 22:49:16.418982 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:16.418944    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9" event={"ID":"5bcc4db3-9814-4b43-a4a8-96f0232c052d","Type":"ContainerStarted","Data":"8b0310916d8ad90056ef7b60b587fef695656f6855e938437afddffa644c5b03"}
Jul 28 22:49:16.419212 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:16.418992    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:16.435557 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:16.435500    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9" podStartSLOduration=1.435482209 podStartE2EDuration="1.435482209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:49:16.433757982 +0000 UTC m=+2358.556105895" watchObservedRunningTime="2026-07-28 22:49:16.435482209 +0000 UTC m=+2358.557830107"
Jul 28 22:49:27.426108 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:27.426062    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9" podUID="5bcc4db3-9814-4b43-a4a8-96f0232c052d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.68:9004: connect: connection refused"
Jul 28 22:49:32.040915 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:32.040882    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:37.425310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:37.425280    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 22:49:39.800339 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:39.800305    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:40.165339 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:40.165252    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:40.478204 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:40.478153    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:40.721026 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:40.720998    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:43.057317 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.057273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:43.150597 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.150559    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:43.156154 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.156126    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"]
Jul 28 22:49:43.159680 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.159659    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.162515 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.162493    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-faf483-serving-cert\""
Jul 28 22:49:43.162627 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.162503    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-89lsd\""
Jul 28 22:49:43.171246 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.171215    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"]
Jul 28 22:49:43.226533 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.226496    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:43.240817 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.240784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff793013-3b7b-4e94-9626-244418ec0d7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.240957 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.240833    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrn4\" (UniqueName: \"kubernetes.io/projected/ff793013-3b7b-4e94-9626-244418ec0d7c-kube-api-access-4xrn4\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.240957 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.240924    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff793013-3b7b-4e94-9626-244418ec0d7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.341978 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.341891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff793013-3b7b-4e94-9626-244418ec0d7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.341978 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.341956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff793013-3b7b-4e94-9626-244418ec0d7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.342213 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.341994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrn4\" (UniqueName: \"kubernetes.io/projected/ff793013-3b7b-4e94-9626-244418ec0d7c-kube-api-access-4xrn4\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.342213 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:49:43.342137    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-serving-cert: secret "maas-api-e2e-shared-b-faf483-serving-cert" not found
Jul 28 22:49:43.342213 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:49:43.342201    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff793013-3b7b-4e94-9626-244418ec0d7c-maas-api-tls podName:ff793013-3b7b-4e94-9626-244418ec0d7c nodeName:}" failed. No retries permitted until 2026-07-28 22:49:43.842183824 +0000 UTC m=+2385.964531700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff793013-3b7b-4e94-9626-244418ec0d7c-maas-api-tls") pod "maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd" (UID: "ff793013-3b7b-4e94-9626-244418ec0d7c") : secret "maas-api-e2e-shared-b-faf483-serving-cert" not found
Jul 28 22:49:43.342547 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.342529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff793013-3b7b-4e94-9626-244418ec0d7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.351392 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.351374    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrn4\" (UniqueName: \"kubernetes.io/projected/ff793013-3b7b-4e94-9626-244418ec0d7c-kube-api-access-4xrn4\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.408151 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.408091    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:43.730308 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.730273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:43.828331 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.828293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:43.848573 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.848542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff793013-3b7b-4e94-9626-244418ec0d7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:43.851078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:43.851048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff793013-3b7b-4e94-9626-244418ec0d7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:44.071507 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:44.071414    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:44.205379 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:44.205352    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"]
Jul 28 22:49:44.207583 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:49:44.207541    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff793013_3b7b_4e94_9626_244418ec0d7c.slice/crio-bdc7adfe4b28d013a2e53401f1f276459432a1ff27449a059fe9052fff82263f WatchSource:0}: Error finding container bdc7adfe4b28d013a2e53401f1f276459432a1ff27449a059fe9052fff82263f: Status 404 returned error can't find the container with id bdc7adfe4b28d013a2e53401f1f276459432a1ff27449a059fe9052fff82263f
Jul 28 22:49:44.566116 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:44.566069    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd" event={"ID":"ff793013-3b7b-4e94-9626-244418ec0d7c","Type":"ContainerStarted","Data":"bdc7adfe4b28d013a2e53401f1f276459432a1ff27449a059fe9052fff82263f"}
Jul 28 22:49:45.572616 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:45.572578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd" event={"ID":"ff793013-3b7b-4e94-9626-244418ec0d7c","Type":"ContainerStarted","Data":"077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf"}
Jul 28 22:49:45.573000 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:45.572652    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:45.589539 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:45.589493    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd" podStartSLOduration=2.206333994 podStartE2EDuration="2.58947775s" totalImagesPullingTime="383.143756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:49:45.5880032 +0000 UTC m=+2387.710351117" watchObservedRunningTime="2026-07-28 22:49:45.58947775 +0000 UTC m=+2387.711825700"
Jul 28 22:49:51.584073 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:51.584039    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 22:49:59.227368 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:59.227332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:49:59.328351 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:49:59.328311    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:50:08.059729 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:08.059686    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:50:22.167194 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.167091    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"]
Jul 28 22:50:22.171068 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.171048    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.175060 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.175019    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a134df\"/\"kube-root-ca.crt\""
Jul 28 22:50:22.175216 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.175019    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a134df\"/\"default-dockercfg-7v695\""
Jul 28 22:50:22.175216 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.175021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a134df\"/\"auth-test-model-a134df-kserve-self-signed-certs\""
Jul 28 22:50:22.175216 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.175066    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a134df\"/\"openshift-service-ca.crt\""
Jul 28 22:50:22.182746 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.182707    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"]
Jul 28 22:50:22.197685 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.197647    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:50:22.215074 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.215030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-dshm\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.215282 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.215151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj2xs\" (UniqueName: \"kubernetes.io/projected/eec1e399-9f37-4142-b082-1bd45c043456-kube-api-access-qj2xs\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.215282 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.215185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-tmp-dir\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.215282 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.215212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec1e399-9f37-4142-b082-1bd45c043456-tls-certs\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.215411 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.215321    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-home\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.215411 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.215403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-model-cache\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.316273 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.316233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec1e399-9f37-4142-b082-1bd45c043456-tls-certs\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.316449 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.316293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-home\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.316449 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.316316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-model-cache\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.316449 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.316358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-dshm\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.316449 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.316396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj2xs\" (UniqueName: \"kubernetes.io/projected/eec1e399-9f37-4142-b082-1bd45c043456-kube-api-access-qj2xs\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.316640 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.316521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-tmp-dir\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.316774 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.316753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-home\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.316892 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.316864    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-model-cache\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.316951 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.316890    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-tmp-dir\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.318750 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.318710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-dshm\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.318922 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.318904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec1e399-9f37-4142-b082-1bd45c043456-tls-certs\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.325012 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.324991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj2xs\" (UniqueName: \"kubernetes.io/projected/eec1e399-9f37-4142-b082-1bd45c043456-kube-api-access-qj2xs\") pod \"auth-test-model-a134df-kserve-6f9db48447-wmfsm\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") " pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.482418 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.482379    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:22.626494 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.626469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"]
Jul 28 22:50:22.628461 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:50:22.628431    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeec1e399_9f37_4142_b082_1bd45c043456.slice/crio-4a24a11d32adff5d6a35c0888635127ded62c0d6da6c53411ba39a92a1c40416 WatchSource:0}: Error finding container 4a24a11d32adff5d6a35c0888635127ded62c0d6da6c53411ba39a92a1c40416: Status 404 returned error can't find the container with id 4a24a11d32adff5d6a35c0888635127ded62c0d6da6c53411ba39a92a1c40416
Jul 28 22:50:22.780869 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.780786    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm" event={"ID":"eec1e399-9f37-4142-b082-1bd45c043456","Type":"ContainerStarted","Data":"0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503"}
Jul 28 22:50:22.780869 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.780828    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm" event={"ID":"eec1e399-9f37-4142-b082-1bd45c043456","Type":"ContainerStarted","Data":"4a24a11d32adff5d6a35c0888635127ded62c0d6da6c53411ba39a92a1c40416"}
Jul 28 22:50:22.795801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:22.795743    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm" podStartSLOduration=0.795719267 podStartE2EDuration="795.719267ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:50:22.795560143 +0000 UTC m=+2424.917908040" watchObservedRunningTime="2026-07-28 22:50:22.795719267 +0000 UTC m=+2424.918067160"
Jul 28 22:50:32.483204 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:32.483169    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:32.483625 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:32.483222    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:32.495549 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:32.495523    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:32.847842 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:32.847757    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:50:36.562327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:36.562293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:50:45.971786 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:45.971746    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"]
Jul 28 22:50:45.975091 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:45.975073    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:45.977738 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:45.977717    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-faf483\"/\"kube-root-ca.crt\""
Jul 28 22:50:45.978831 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:45.978807    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-faf483\"/\"auth-test-model-faf483-kserve-self-signed-certs\""
Jul 28 22:50:45.978989 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:45.978972    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-faf483\"/\"default-dockercfg-fnzn6\""
Jul 28 22:50:45.979038 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:45.978997    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-faf483\"/\"openshift-service-ca.crt\""
Jul 28 22:50:45.982908 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:45.982874    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"]
Jul 28 22:50:45.994825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:45.994791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:50:46.152616 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.152580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnqw9\" (UniqueName: \"kubernetes.io/projected/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-kube-api-access-cnqw9\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.152787 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.152700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tmp-dir\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.152787 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.152730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-dshm\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.152787 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.152751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tls-certs\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.152953 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.152788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-model-cache\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.152953 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.152862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-home\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.253604 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.253522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tmp-dir\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.253604 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.253557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-dshm\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.253604 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.253575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tls-certs\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.253875 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.253697    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-model-cache\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.253875 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.253731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-home\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.253875 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.253805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnqw9\" (UniqueName: \"kubernetes.io/projected/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-kube-api-access-cnqw9\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.254054 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.254030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tmp-dir\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.254150 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.254073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-model-cache\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.254202 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.254155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-home\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.256178 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.256154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-dshm\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.256362 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.256341    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tls-certs\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.261220 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.261190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnqw9\" (UniqueName: \"kubernetes.io/projected/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-kube-api-access-cnqw9\") pod \"auth-test-model-faf483-kserve-f9cfc944c-8njg2\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") " pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.286317 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.286285    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:46.420896 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.420859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"]
Jul 28 22:50:46.422628 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:50:46.422602    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27cf9062_1db4_4488_afcf_c1cd1eb5a7bf.slice/crio-c6fcc1254ce442dc633e1ce05807fe03b23879483c03420afd495aa5496c45df WatchSource:0}: Error finding container c6fcc1254ce442dc633e1ce05807fe03b23879483c03420afd495aa5496c45df: Status 404 returned error can't find the container with id c6fcc1254ce442dc633e1ce05807fe03b23879483c03420afd495aa5496c45df
Jul 28 22:50:46.912619 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.912581    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2" event={"ID":"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf","Type":"ContainerStarted","Data":"70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0"}
Jul 28 22:50:46.912794 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.912627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2" event={"ID":"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf","Type":"ContainerStarted","Data":"c6fcc1254ce442dc633e1ce05807fe03b23879483c03420afd495aa5496c45df"}
Jul 28 22:50:46.932816 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:46.932755    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2" podStartSLOduration=1.9327390740000001 podStartE2EDuration="1.932739074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:50:46.931337867 +0000 UTC m=+2449.053685762" watchObservedRunningTime="2026-07-28 22:50:46.932739074 +0000 UTC m=+2449.055086967"
Jul 28 22:50:56.287088 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:56.287038    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:56.289735 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:56.287122    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:56.299491 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:56.299468    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:50:56.976659 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:50:56.976631    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:51:00.927780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:51:00.927729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:51:01.028433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:51:01.028394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:51:55.961156 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:51:55.961058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:51:58.685461 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:51:58.685430    2560 scope.go:117] "RemoveContainer" containerID="76822fdcc04a087bd15ba5c870a36c024f8510a2d8393319a3123cad9a3318e2"
Jul 28 22:53:58.714545 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:53:58.714471    2560 scope.go:117] "RemoveContainer" containerID="c08cb4409c2927ea9ea7b9c30447ad8a60818aafdf794ed51b7a59c70cfb3122"
Jul 28 22:54:18.696572 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:54:18.696536    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:54:18.748999 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:54:18.748969    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:54:58.735209 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:54:58.735120    2560 scope.go:117] "RemoveContainer" containerID="90988a2b7b251f334fb7fad2ad653f67060197a4ea32ba8f76afac665d908b2a"
Jul 28 22:54:58.743741 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:54:58.743722    2560 scope.go:117] "RemoveContainer" containerID="5756e07f007665ecee3b9d38cbc7cf49b03381f146f117cf678d426ccbf1125f"
Jul 28 22:55:02.675988 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:02.675955    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"]
Jul 28 22:55:02.689367 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:02.689327    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:55:03.577574 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:03.577541    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"]
Jul 28 22:55:03.595476 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:03.595437    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:55:14.695245 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:14.695210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:55:17.695534 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:17.695472    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm" podUID="eec1e399-9f37-4142-b082-1bd45c043456" containerName="main" containerID="cri-o://0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503" gracePeriod=45
Jul 28 22:55:17.951131 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:17.951048    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:55:18.033863 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.033825    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj2xs\" (UniqueName: \"kubernetes.io/projected/eec1e399-9f37-4142-b082-1bd45c043456-kube-api-access-qj2xs\") pod \"eec1e399-9f37-4142-b082-1bd45c043456\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") "
Jul 28 22:55:18.034039 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.033937    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-tmp-dir\") pod \"eec1e399-9f37-4142-b082-1bd45c043456\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") "
Jul 28 22:55:18.034039 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.033978    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-dshm\") pod \"eec1e399-9f37-4142-b082-1bd45c043456\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") "
Jul 28 22:55:18.034039 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.034026    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-home\") pod \"eec1e399-9f37-4142-b082-1bd45c043456\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") "
Jul 28 22:55:18.034226 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.034054    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec1e399-9f37-4142-b082-1bd45c043456-tls-certs\") pod \"eec1e399-9f37-4142-b082-1bd45c043456\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") "
Jul 28 22:55:18.034226 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.034080    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-model-cache\") pod \"eec1e399-9f37-4142-b082-1bd45c043456\" (UID: \"eec1e399-9f37-4142-b082-1bd45c043456\") "
Jul 28 22:55:18.034328 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.034250    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eec1e399-9f37-4142-b082-1bd45c043456" (UID: "eec1e399-9f37-4142-b082-1bd45c043456"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:55:18.034328 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.034296    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-home" (OuterVolumeSpecName: "home") pod "eec1e399-9f37-4142-b082-1bd45c043456" (UID: "eec1e399-9f37-4142-b082-1bd45c043456"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:55:18.034458 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.034429    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-model-cache" (OuterVolumeSpecName: "model-cache") pod "eec1e399-9f37-4142-b082-1bd45c043456" (UID: "eec1e399-9f37-4142-b082-1bd45c043456"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:55:18.034524 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.034463    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:18.034524 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.034483    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:18.036252 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.036223    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec1e399-9f37-4142-b082-1bd45c043456-kube-api-access-qj2xs" (OuterVolumeSpecName: "kube-api-access-qj2xs") pod "eec1e399-9f37-4142-b082-1bd45c043456" (UID: "eec1e399-9f37-4142-b082-1bd45c043456"). InnerVolumeSpecName "kube-api-access-qj2xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:55:18.036375 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.036293    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec1e399-9f37-4142-b082-1bd45c043456-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eec1e399-9f37-4142-b082-1bd45c043456" (UID: "eec1e399-9f37-4142-b082-1bd45c043456"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:55:18.036375 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.036307    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-dshm" (OuterVolumeSpecName: "dshm") pod "eec1e399-9f37-4142-b082-1bd45c043456" (UID: "eec1e399-9f37-4142-b082-1bd45c043456"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:55:18.135444 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.135408    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:18.135444 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.135436    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec1e399-9f37-4142-b082-1bd45c043456-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:18.135444 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.135446    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec1e399-9f37-4142-b082-1bd45c043456-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:18.135668 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.135456    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj2xs\" (UniqueName: \"kubernetes.io/projected/eec1e399-9f37-4142-b082-1bd45c043456-kube-api-access-qj2xs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:18.338502 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.338469    2560 generic.go:356] "Generic (PLEG): container finished" podID="eec1e399-9f37-4142-b082-1bd45c043456" containerID="0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503" exitCode=0
Jul 28 22:55:18.338682 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.338514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm" event={"ID":"eec1e399-9f37-4142-b082-1bd45c043456","Type":"ContainerDied","Data":"0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503"}
Jul 28 22:55:18.338682 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.338543    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm" event={"ID":"eec1e399-9f37-4142-b082-1bd45c043456","Type":"ContainerDied","Data":"4a24a11d32adff5d6a35c0888635127ded62c0d6da6c53411ba39a92a1c40416"}
Jul 28 22:55:18.338682 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.338560    2560 scope.go:117] "RemoveContainer" containerID="0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503"
Jul 28 22:55:18.338682 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.338515    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"
Jul 28 22:55:18.351580 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.351558    2560 scope.go:117] "RemoveContainer" containerID="0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503"
Jul 28 22:55:18.351942 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:55:18.351920    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503\": container with ID starting with 0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503 not found: ID does not exist" containerID="0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503"
Jul 28 22:55:18.351997 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.351951    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503"} err="failed to get container status \"0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503\": rpc error: code = NotFound desc = could not find container \"0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503\": container with ID starting with 0368d594a733ef442d2c64d854cc78b81aa3f9e87f101c915a1873de75ff5503 not found: ID does not exist"
Jul 28 22:55:18.359886 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.359859    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"]
Jul 28 22:55:18.365799 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.365770    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/auth-test-model-a134df-kserve-6f9db48447-wmfsm"]
Jul 28 22:55:18.407696 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.407661    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec1e399-9f37-4142-b082-1bd45c043456" path="/var/lib/kubelet/pods/eec1e399-9f37-4142-b082-1bd45c043456/volumes"
Jul 28 22:55:18.598083 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.597995    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2" podUID="27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" containerName="main" containerID="cri-o://70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0" gracePeriod=45
Jul 28 22:55:18.845643 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.845613    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:55:18.943817 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.943732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-model-cache\") pod \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") "
Jul 28 22:55:18.943817 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.943765    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-dshm\") pod \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") "
Jul 28 22:55:18.943817 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.943811    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnqw9\" (UniqueName: \"kubernetes.io/projected/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-kube-api-access-cnqw9\") pod \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") "
Jul 28 22:55:18.944094 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.943842    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tmp-dir\") pod \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") "
Jul 28 22:55:18.944094 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.943861    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tls-certs\") pod \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") "
Jul 28 22:55:18.944094 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.943877    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-home\") pod \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\" (UID: \"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf\") "
Jul 28 22:55:18.944309 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.944066    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" (UID: "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:55:18.944309 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.944214    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" (UID: "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:55:18.944433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.944306    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:18.944433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.944333    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:18.944433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.944298    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-home" (OuterVolumeSpecName: "home") pod "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" (UID: "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:55:18.946127 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.946089    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-dshm" (OuterVolumeSpecName: "dshm") pod "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" (UID: "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:55:18.946236 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.946128    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" (UID: "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:55:18.946236 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:18.946130    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-kube-api-access-cnqw9" (OuterVolumeSpecName: "kube-api-access-cnqw9") pod "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" (UID: "27cf9062-1db4-4488-afcf-c1cd1eb5a7bf"). InnerVolumeSpecName "kube-api-access-cnqw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:55:19.045643 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.045605    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:19.045643 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.045639    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnqw9\" (UniqueName: \"kubernetes.io/projected/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-kube-api-access-cnqw9\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:19.045643 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.045650    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:19.045882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.045659    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:55:19.345587 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.345553    2560 generic.go:356] "Generic (PLEG): container finished" podID="27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" containerID="70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0" exitCode=0
Jul 28 22:55:19.345756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.345596    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"
Jul 28 22:55:19.345756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.345650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2" event={"ID":"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf","Type":"ContainerDied","Data":"70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0"}
Jul 28 22:55:19.345756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.345699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2" event={"ID":"27cf9062-1db4-4488-afcf-c1cd1eb5a7bf","Type":"ContainerDied","Data":"c6fcc1254ce442dc633e1ce05807fe03b23879483c03420afd495aa5496c45df"}
Jul 28 22:55:19.345756 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.345722    2560 scope.go:117] "RemoveContainer" containerID="70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0"
Jul 28 22:55:19.355310 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.355295    2560 scope.go:117] "RemoveContainer" containerID="70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0"
Jul 28 22:55:19.355544 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:55:19.355525    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0\": container with ID starting with 70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0 not found: ID does not exist" containerID="70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0"
Jul 28 22:55:19.355595 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.355553    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0"} err="failed to get container status \"70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0\": rpc error: code = NotFound desc = could not find container \"70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0\": container with ID starting with 70dda019276e952b719cb006c14a4fb32b9567ffaa2587987e7bb171663647e0 not found: ID does not exist"
Jul 28 22:55:19.366677 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.366656    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"]
Jul 28 22:55:19.369529 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:19.369507    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/auth-test-model-faf483-kserve-f9cfc944c-8njg2"]
Jul 28 22:55:20.407556 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:20.407523    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" path="/var/lib/kubelet/pods/27cf9062-1db4-4488-afcf-c1cd1eb5a7bf/volumes"
Jul 28 22:55:34.900134 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:34.900076    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:55:57.067758 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.067715    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"]
Jul 28 22:55:57.070306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.068308    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec1e399-9f37-4142-b082-1bd45c043456" containerName="main"
Jul 28 22:55:57.070306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.068325    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec1e399-9f37-4142-b082-1bd45c043456" containerName="main"
Jul 28 22:55:57.070306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.068346    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" containerName="main"
Jul 28 22:55:57.070306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.068352    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" containerName="main"
Jul 28 22:55:57.070306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.068420    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec1e399-9f37-4142-b082-1bd45c043456" containerName="main"
Jul 28 22:55:57.070306 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.068432    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cf9062-1db4-4488-afcf-c1cd1eb5a7bf" containerName="main"
Jul 28 22:55:57.082628 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.082598    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"]
Jul 28 22:55:57.082794 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.082718    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.086229 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.086207    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-faf483\"/\"default-dockercfg-fnzn6\""
Jul 28 22:55:57.086353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.086259    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-faf483\"/\"openshift-service-ca.crt\""
Jul 28 22:55:57.086353 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.086292    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-faf483\"/\"kube-root-ca.crt\""
Jul 28 22:55:57.086452 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.086353    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-faf483\"/\"sub-test-model-faf483-kserve-self-signed-certs\""
Jul 28 22:55:57.096481 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.096457    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:55:57.210339 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.210305    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzzv\" (UniqueName: \"kubernetes.io/projected/ea76d584-f5b9-4c78-b605-0407fbce43b5-kube-api-access-hvzzv\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.210529 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.210363    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-model-cache\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.210529 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.210410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea76d584-f5b9-4c78-b605-0407fbce43b5-tls-certs\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.210529 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.210442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-dshm\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.210996 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.210520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-home\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.210996 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.210578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-tmp-dir\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.311874 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.311838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-dshm\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.311874 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.311878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-home\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.312081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.311904    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-tmp-dir\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.312081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.311944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzzv\" (UniqueName: \"kubernetes.io/projected/ea76d584-f5b9-4c78-b605-0407fbce43b5-kube-api-access-hvzzv\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.312081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.311979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-model-cache\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.312081 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.312010    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea76d584-f5b9-4c78-b605-0407fbce43b5-tls-certs\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.312366 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.312326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-home\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.312479 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.312389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-tmp-dir\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.312479 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.312436    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-model-cache\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.314230 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.314213    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-dshm\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.314484 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.314468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea76d584-f5b9-4c78-b605-0407fbce43b5-tls-certs\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.319825 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.319770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzzv\" (UniqueName: \"kubernetes.io/projected/ea76d584-f5b9-4c78-b605-0407fbce43b5-kube-api-access-hvzzv\") pod \"sub-test-model-faf483-kserve-79d5757675-jgjk4\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") " pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.393887 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.393847    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:55:57.526449 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.526420    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"]
Jul 28 22:55:57.528659 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:55:57.528634    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea76d584_f5b9_4c78_b605_0407fbce43b5.slice/crio-e3afb42cca7889bec59a2093ff9e0334182db422c9a4c9d1418f2c77958aa55e WatchSource:0}: Error finding container e3afb42cca7889bec59a2093ff9e0334182db422c9a4c9d1418f2c77958aa55e: Status 404 returned error can't find the container with id e3afb42cca7889bec59a2093ff9e0334182db422c9a4c9d1418f2c77958aa55e
Jul 28 22:55:57.550056 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:57.550023    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4" event={"ID":"ea76d584-f5b9-4c78-b605-0407fbce43b5","Type":"ContainerStarted","Data":"e3afb42cca7889bec59a2093ff9e0334182db422c9a4c9d1418f2c77958aa55e"}
Jul 28 22:55:58.556800 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:58.556755    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4" event={"ID":"ea76d584-f5b9-4c78-b605-0407fbce43b5","Type":"ContainerStarted","Data":"a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a"}
Jul 28 22:55:58.573965 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:55:58.573918    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4" podStartSLOduration=1.5739048690000002 podStartE2EDuration="1.573904869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:55:58.57178474 +0000 UTC m=+2760.694132633" watchObservedRunningTime="2026-07-28 22:55:58.573904869 +0000 UTC m=+2760.696252762"
Jul 28 22:56:07.394744 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:56:07.394699    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:56:07.395205 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:56:07.394760    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:56:07.407714 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:56:07.407685    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:56:07.614740 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:56:07.614703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:56:57.773234 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:56:57.773149    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:56:58.579696 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:56:58.579659    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"]
Jul 28 22:56:59.042566 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:56:59.042527    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:57:02.476663 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.476632    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"]
Jul 28 22:57:02.480307 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.480290    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.483400 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.483379    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a134df\"/\"openshift-service-ca.crt\""
Jul 28 22:57:02.484801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.484780    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a134df\"/\"default-dockercfg-7v695\""
Jul 28 22:57:02.484916 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.484801    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a134df\"/\"kube-root-ca.crt\""
Jul 28 22:57:02.484916 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.484783    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a134df\"/\"rate-test-model-a134df-kserve-self-signed-certs\""
Jul 28 22:57:02.491231 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.491205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"]
Jul 28 22:57:02.544321 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.544287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-model-cache\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.544499 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.544367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-dshm\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.544499 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.544393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-home\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.544499 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.544440    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec65986-7061-4fbd-b6ed-87c1d8f03705-tls-certs\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.544499 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.544476    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-tmp-dir\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.544692 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.544499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75x2\" (UniqueName: \"kubernetes.io/projected/dec65986-7061-4fbd-b6ed-87c1d8f03705-kube-api-access-t75x2\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.645969 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.645935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec65986-7061-4fbd-b6ed-87c1d8f03705-tls-certs\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.645969 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.645980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-tmp-dir\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.646280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.646003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75x2\" (UniqueName: \"kubernetes.io/projected/dec65986-7061-4fbd-b6ed-87c1d8f03705-kube-api-access-t75x2\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.646280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.646047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-model-cache\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.646280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.646145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-dshm\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.646280 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.646176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-home\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.646504 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.646432    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-tmp-dir\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.646504 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.646477    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-model-cache\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.646504 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.646493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-home\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.648430 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.648403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-dshm\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.648735 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.648716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec65986-7061-4fbd-b6ed-87c1d8f03705-tls-certs\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.655338 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.655320    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75x2\" (UniqueName: \"kubernetes.io/projected/dec65986-7061-4fbd-b6ed-87c1d8f03705-kube-api-access-t75x2\") pod \"rate-test-model-a134df-kserve-5479dcccd7-g4cg4\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") " pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.789993 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.789897    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:02.933823 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.933795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"]
Jul 28 22:57:02.936220 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:57:02.936192    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddec65986_7061_4fbd_b6ed_87c1d8f03705.slice/crio-32ccae395c1b8dff7ba8f928b3663a066220024266943262b59694bd7e9b6a2c WatchSource:0}: Error finding container 32ccae395c1b8dff7ba8f928b3663a066220024266943262b59694bd7e9b6a2c: Status 404 returned error can't find the container with id 32ccae395c1b8dff7ba8f928b3663a066220024266943262b59694bd7e9b6a2c
Jul 28 22:57:02.974311 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:02.974271    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:57:03.892789 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:03.892754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4" event={"ID":"dec65986-7061-4fbd-b6ed-87c1d8f03705","Type":"ContainerStarted","Data":"3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0"}
Jul 28 22:57:03.893372 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:03.892797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4" event={"ID":"dec65986-7061-4fbd-b6ed-87c1d8f03705","Type":"ContainerStarted","Data":"32ccae395c1b8dff7ba8f928b3663a066220024266943262b59694bd7e9b6a2c"}
Jul 28 22:57:03.914619 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:03.914571    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4" podStartSLOduration=1.9145556190000002 podStartE2EDuration="1.914555619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:57:03.912041275 +0000 UTC m=+2826.034389168" watchObservedRunningTime="2026-07-28 22:57:03.914555619 +0000 UTC m=+2826.036903509"
Jul 28 22:57:08.177791 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:08.177752    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:57:08.279829 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:08.279797    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:57:12.790114 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:12.790065    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:12.790597 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:12.790145    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:12.803188 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:12.803162    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:12.953241 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:12.953159    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:57:13.599170 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.599125    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4" podUID="ea76d584-f5b9-4c78-b605-0407fbce43b5" containerName="main" containerID="cri-o://a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a" gracePeriod=45
Jul 28 22:57:13.854574 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.854502    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:57:13.947850 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.947816    2560 generic.go:356] "Generic (PLEG): container finished" podID="ea76d584-f5b9-4c78-b605-0407fbce43b5" containerID="a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a" exitCode=0
Jul 28 22:57:13.948046 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.947860    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"
Jul 28 22:57:13.948046 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.947903    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4" event={"ID":"ea76d584-f5b9-4c78-b605-0407fbce43b5","Type":"ContainerDied","Data":"a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a"}
Jul 28 22:57:13.948046 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.947947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4" event={"ID":"ea76d584-f5b9-4c78-b605-0407fbce43b5","Type":"ContainerDied","Data":"e3afb42cca7889bec59a2093ff9e0334182db422c9a4c9d1418f2c77958aa55e"}
Jul 28 22:57:13.948046 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.947967    2560 scope.go:117] "RemoveContainer" containerID="a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a"
Jul 28 22:57:13.956686 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.956665    2560 scope.go:117] "RemoveContainer" containerID="a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a"
Jul 28 22:57:13.956959 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:57:13.956935    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a\": container with ID starting with a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a not found: ID does not exist" containerID="a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a"
Jul 28 22:57:13.957014 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.956971    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a"} err="failed to get container status \"a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a\": rpc error: code = NotFound desc = could not find container \"a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a\": container with ID starting with a684f15da1cb81e12b0454eb3dd8620987fa48175f6b39c7da1a5944fd62009a not found: ID does not exist"
Jul 28 22:57:13.962348 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962325    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-dshm\") pod \"ea76d584-f5b9-4c78-b605-0407fbce43b5\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") "
Jul 28 22:57:13.962463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962365    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-model-cache\") pod \"ea76d584-f5b9-4c78-b605-0407fbce43b5\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") "
Jul 28 22:57:13.962463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962413    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea76d584-f5b9-4c78-b605-0407fbce43b5-tls-certs\") pod \"ea76d584-f5b9-4c78-b605-0407fbce43b5\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") "
Jul 28 22:57:13.962463 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-home\") pod \"ea76d584-f5b9-4c78-b605-0407fbce43b5\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") "
Jul 28 22:57:13.962623 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962498    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvzzv\" (UniqueName: \"kubernetes.io/projected/ea76d584-f5b9-4c78-b605-0407fbce43b5-kube-api-access-hvzzv\") pod \"ea76d584-f5b9-4c78-b605-0407fbce43b5\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") "
Jul 28 22:57:13.962623 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962542    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-tmp-dir\") pod \"ea76d584-f5b9-4c78-b605-0407fbce43b5\" (UID: \"ea76d584-f5b9-4c78-b605-0407fbce43b5\") "
Jul 28 22:57:13.962726 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962685    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea76d584-f5b9-4c78-b605-0407fbce43b5" (UID: "ea76d584-f5b9-4c78-b605-0407fbce43b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:57:13.962726 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962700    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-home" (OuterVolumeSpecName: "home") pod "ea76d584-f5b9-4c78-b605-0407fbce43b5" (UID: "ea76d584-f5b9-4c78-b605-0407fbce43b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:57:13.962828 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962805    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea76d584-f5b9-4c78-b605-0407fbce43b5" (UID: "ea76d584-f5b9-4c78-b605-0407fbce43b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:57:13.962948 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962933    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:57:13.962995 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962951    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:57:13.962995 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.962960    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:57:13.964525 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.964496    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea76d584-f5b9-4c78-b605-0407fbce43b5-kube-api-access-hvzzv" (OuterVolumeSpecName: "kube-api-access-hvzzv") pod "ea76d584-f5b9-4c78-b605-0407fbce43b5" (UID: "ea76d584-f5b9-4c78-b605-0407fbce43b5"). InnerVolumeSpecName "kube-api-access-hvzzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:57:13.964937 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.964913    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-dshm" (OuterVolumeSpecName: "dshm") pod "ea76d584-f5b9-4c78-b605-0407fbce43b5" (UID: "ea76d584-f5b9-4c78-b605-0407fbce43b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:57:13.964937 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:13.964914    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea76d584-f5b9-4c78-b605-0407fbce43b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea76d584-f5b9-4c78-b605-0407fbce43b5" (UID: "ea76d584-f5b9-4c78-b605-0407fbce43b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:57:14.064433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:14.064391    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea76d584-f5b9-4c78-b605-0407fbce43b5-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:57:14.064433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:14.064422    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea76d584-f5b9-4c78-b605-0407fbce43b5-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:57:14.064433 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:14.064434    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzzv\" (UniqueName: \"kubernetes.io/projected/ea76d584-f5b9-4c78-b605-0407fbce43b5-kube-api-access-hvzzv\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:57:14.272980 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:14.272943    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"]
Jul 28 22:57:14.274851 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:14.274829    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-faf483/sub-test-model-faf483-kserve-79d5757675-jgjk4"]
Jul 28 22:57:14.407890 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:14.407857    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea76d584-f5b9-4c78-b605-0407fbce43b5" path="/var/lib/kubelet/pods/ea76d584-f5b9-4c78-b605-0407fbce43b5/volumes"
Jul 28 22:57:38.181319 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:57:38.181279    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:58:44.750362 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:44.750328    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"]
Jul 28 22:58:44.770034 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:44.770004    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:58:45.572807 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:45.572773    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:58:46.515477 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.515438    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7413-140606480635184"]
Jul 28 22:58:46.516057 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.516036    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea76d584-f5b9-4c78-b605-0407fbce43b5" containerName="main"
Jul 28 22:58:46.516180 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.516060    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea76d584-f5b9-4c78-b605-0407fbce43b5" containerName="main"
Jul 28 22:58:46.516243 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.516233    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea76d584-f5b9-4c78-b605-0407fbce43b5" containerName="main"
Jul 28 22:58:46.519364 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.519343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480635184"
Jul 28 22:58:46.521959 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.521937    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vq2lx\""
Jul 28 22:58:46.524443 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.524420    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480635184"]
Jul 28 22:58:46.621694 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.621660    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rbc\" (UniqueName: \"kubernetes.io/projected/a5a7a86a-f291-48bc-bd51-b4be91ca30f0-kube-api-access-j2rbc\") pod \"test-curl-7413-140606480635184\" (UID: \"a5a7a86a-f291-48bc-bd51-b4be91ca30f0\") " pod="opendatahub/test-curl-7413-140606480635184"
Jul 28 22:58:46.722905 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.722872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rbc\" (UniqueName: \"kubernetes.io/projected/a5a7a86a-f291-48bc-bd51-b4be91ca30f0-kube-api-access-j2rbc\") pod \"test-curl-7413-140606480635184\" (UID: \"a5a7a86a-f291-48bc-bd51-b4be91ca30f0\") " pod="opendatahub/test-curl-7413-140606480635184"
Jul 28 22:58:46.730496 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.730468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rbc\" (UniqueName: \"kubernetes.io/projected/a5a7a86a-f291-48bc-bd51-b4be91ca30f0-kube-api-access-j2rbc\") pod \"test-curl-7413-140606480635184\" (UID: \"a5a7a86a-f291-48bc-bd51-b4be91ca30f0\") " pod="opendatahub/test-curl-7413-140606480635184"
Jul 28 22:58:46.829929 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.829850    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480635184"
Jul 28 22:58:46.961959 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.961934    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480635184"]
Jul 28 22:58:46.964307 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:58:46.964274    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5a7a86a_f291_48bc_bd51_b4be91ca30f0.slice/crio-4fb335e335ab9c9b019ecab48de2e0cd493c0ec64e1def816c16e76a2a018b72 WatchSource:0}: Error finding container 4fb335e335ab9c9b019ecab48de2e0cd493c0ec64e1def816c16e76a2a018b72: Status 404 returned error can't find the container with id 4fb335e335ab9c9b019ecab48de2e0cd493c0ec64e1def816c16e76a2a018b72
Jul 28 22:58:46.965790 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:46.965771    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:58:47.429065 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:47.429031    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480635184" event={"ID":"a5a7a86a-f291-48bc-bd51-b4be91ca30f0","Type":"ContainerStarted","Data":"4fb335e335ab9c9b019ecab48de2e0cd493c0ec64e1def816c16e76a2a018b72"}
Jul 28 22:58:48.435946 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:48.435905    2560 generic.go:356] "Generic (PLEG): container finished" podID="a5a7a86a-f291-48bc-bd51-b4be91ca30f0" containerID="33a32dd573ff330793757d83438671d5745a5b8aea5d4bc64acdac234d42084e" exitCode=0
Jul 28 22:58:48.436344 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:48.435966    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480635184" event={"ID":"a5a7a86a-f291-48bc-bd51-b4be91ca30f0","Type":"ContainerDied","Data":"33a32dd573ff330793757d83438671d5745a5b8aea5d4bc64acdac234d42084e"}
Jul 28 22:58:49.578388 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.578362    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480635184"
Jul 28 22:58:49.597955 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.597921    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7413-140606480635184_a5a7a86a-f291-48bc-bd51-b4be91ca30f0/test-curl-7413-140606480635184/0.log"
Jul 28 22:58:49.623362 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.623332    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7413-140606480635184"]
Jul 28 22:58:49.628216 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.628193    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7413-140606480635184"]
Jul 28 22:58:49.653749 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.653725    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2rbc\" (UniqueName: \"kubernetes.io/projected/a5a7a86a-f291-48bc-bd51-b4be91ca30f0-kube-api-access-j2rbc\") pod \"a5a7a86a-f291-48bc-bd51-b4be91ca30f0\" (UID: \"a5a7a86a-f291-48bc-bd51-b4be91ca30f0\") "
Jul 28 22:58:49.655897 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.655876    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a7a86a-f291-48bc-bd51-b4be91ca30f0-kube-api-access-j2rbc" (OuterVolumeSpecName: "kube-api-access-j2rbc") pod "a5a7a86a-f291-48bc-bd51-b4be91ca30f0" (UID: "a5a7a86a-f291-48bc-bd51-b4be91ca30f0"). InnerVolumeSpecName "kube-api-access-j2rbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:58:49.747766 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.747732    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7413-140606480650928"]
Jul 28 22:58:49.748161 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.748147    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a7a86a-f291-48bc-bd51-b4be91ca30f0" containerName="test-curl-7413-140606480635184"
Jul 28 22:58:49.748210 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.748162    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a7a86a-f291-48bc-bd51-b4be91ca30f0" containerName="test-curl-7413-140606480635184"
Jul 28 22:58:49.748288 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.748233    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a7a86a-f291-48bc-bd51-b4be91ca30f0" containerName="test-curl-7413-140606480635184"
Jul 28 22:58:49.751449 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.751433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480650928"
Jul 28 22:58:49.754839 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.754812    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2rbc\" (UniqueName: \"kubernetes.io/projected/a5a7a86a-f291-48bc-bd51-b4be91ca30f0-kube-api-access-j2rbc\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:58:49.756532 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.756511    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480650928"]
Jul 28 22:58:49.855886 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.855852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbgb\" (UniqueName: \"kubernetes.io/projected/30dc4cbe-c56e-4c5c-8ae3-2c87033f220c-kube-api-access-mvbgb\") pod \"test-curl-7413-140606480650928\" (UID: \"30dc4cbe-c56e-4c5c-8ae3-2c87033f220c\") " pod="opendatahub/test-curl-7413-140606480650928"
Jul 28 22:58:49.957322 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.957295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbgb\" (UniqueName: \"kubernetes.io/projected/30dc4cbe-c56e-4c5c-8ae3-2c87033f220c-kube-api-access-mvbgb\") pod \"test-curl-7413-140606480650928\" (UID: \"30dc4cbe-c56e-4c5c-8ae3-2c87033f220c\") " pod="opendatahub/test-curl-7413-140606480650928"
Jul 28 22:58:49.964785 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:49.964763    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbgb\" (UniqueName: \"kubernetes.io/projected/30dc4cbe-c56e-4c5c-8ae3-2c87033f220c-kube-api-access-mvbgb\") pod \"test-curl-7413-140606480650928\" (UID: \"30dc4cbe-c56e-4c5c-8ae3-2c87033f220c\") " pod="opendatahub/test-curl-7413-140606480650928"
Jul 28 22:58:50.061579 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:50.061514    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480650928"
Jul 28 22:58:50.192652 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:50.192628    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480650928"]
Jul 28 22:58:50.194310 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:58:50.194288    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30dc4cbe_c56e_4c5c_8ae3_2c87033f220c.slice/crio-17b58cf48528f2c7aadc1d6fa947778467ebdcf3daccce67f545b22205194728 WatchSource:0}: Error finding container 17b58cf48528f2c7aadc1d6fa947778467ebdcf3daccce67f545b22205194728: Status 404 returned error can't find the container with id 17b58cf48528f2c7aadc1d6fa947778467ebdcf3daccce67f545b22205194728
Jul 28 22:58:50.407444 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:50.407359    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a7a86a-f291-48bc-bd51-b4be91ca30f0" path="/var/lib/kubelet/pods/a5a7a86a-f291-48bc-bd51-b4be91ca30f0/volumes"
Jul 28 22:58:50.446983 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:50.446956    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480635184"
Jul 28 22:58:50.446983 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:50.446979    2560 scope.go:117] "RemoveContainer" containerID="33a32dd573ff330793757d83438671d5745a5b8aea5d4bc64acdac234d42084e"
Jul 28 22:58:50.448461 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:50.448429    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480650928" event={"ID":"30dc4cbe-c56e-4c5c-8ae3-2c87033f220c","Type":"ContainerStarted","Data":"17b58cf48528f2c7aadc1d6fa947778467ebdcf3daccce67f545b22205194728"}
Jul 28 22:58:51.455395 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:51.455363    2560 generic.go:356] "Generic (PLEG): container finished" podID="30dc4cbe-c56e-4c5c-8ae3-2c87033f220c" containerID="8f70283b452b1c311bcfef3ba339db2ac6d7d824b360b5b71b108d76da4e1326" exitCode=0
Jul 28 22:58:51.455775 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:51.455401    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480650928" event={"ID":"30dc4cbe-c56e-4c5c-8ae3-2c87033f220c","Type":"ContainerDied","Data":"8f70283b452b1c311bcfef3ba339db2ac6d7d824b360b5b71b108d76da4e1326"}
Jul 28 22:58:52.595720 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.595695    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480650928"
Jul 28 22:58:52.614882 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.614857    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7413-140606480650928_30dc4cbe-c56e-4c5c-8ae3-2c87033f220c/test-curl-7413-140606480650928/0.log"
Jul 28 22:58:52.638581 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.638552    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7413-140606480650928"]
Jul 28 22:58:52.640451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.640429    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7413-140606480650928"]
Jul 28 22:58:52.683912 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.683882    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvbgb\" (UniqueName: \"kubernetes.io/projected/30dc4cbe-c56e-4c5c-8ae3-2c87033f220c-kube-api-access-mvbgb\") pod \"30dc4cbe-c56e-4c5c-8ae3-2c87033f220c\" (UID: \"30dc4cbe-c56e-4c5c-8ae3-2c87033f220c\") "
Jul 28 22:58:52.685884 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.685856    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30dc4cbe-c56e-4c5c-8ae3-2c87033f220c-kube-api-access-mvbgb" (OuterVolumeSpecName: "kube-api-access-mvbgb") pod "30dc4cbe-c56e-4c5c-8ae3-2c87033f220c" (UID: "30dc4cbe-c56e-4c5c-8ae3-2c87033f220c"). InnerVolumeSpecName "kube-api-access-mvbgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:58:52.785176 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.785145    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvbgb\" (UniqueName: \"kubernetes.io/projected/30dc4cbe-c56e-4c5c-8ae3-2c87033f220c-kube-api-access-mvbgb\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:58:52.794322 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.794294    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:52.794727 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.794708    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30dc4cbe-c56e-4c5c-8ae3-2c87033f220c" containerName="test-curl-7413-140606480650928"
Jul 28 22:58:52.794773 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.794730    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30dc4cbe-c56e-4c5c-8ae3-2c87033f220c" containerName="test-curl-7413-140606480650928"
Jul 28 22:58:52.794814 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.794796    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30dc4cbe-c56e-4c5c-8ae3-2c87033f220c" containerName="test-curl-7413-140606480650928"
Jul 28 22:58:52.797793 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.797776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:52.803741 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.803718    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:52.886576 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.886544    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rhx\" (UniqueName: \"kubernetes.io/projected/9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb-kube-api-access-n4rhx\") pod \"test-curl-7413-140606480672784\" (UID: \"9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb\") " pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:52.987964 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.987929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rhx\" (UniqueName: \"kubernetes.io/projected/9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb-kube-api-access-n4rhx\") pod \"test-curl-7413-140606480672784\" (UID: \"9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb\") " pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:52.995885 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:52.995861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rhx\" (UniqueName: \"kubernetes.io/projected/9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb-kube-api-access-n4rhx\") pod \"test-curl-7413-140606480672784\" (UID: \"9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb\") " pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:53.108765 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:53.108679    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:53.452455 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:53.452426    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:53.455044 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:58:53.455020    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f5ae2f5_d5fe_497f_88d9_c65a4fdadccb.slice/crio-b61d000d60a3b5f18fc1d94a32ba64ac2c1ed02c87e77dedd9cb604d2996e60e WatchSource:0}: Error finding container b61d000d60a3b5f18fc1d94a32ba64ac2c1ed02c87e77dedd9cb604d2996e60e: Status 404 returned error can't find the container with id b61d000d60a3b5f18fc1d94a32ba64ac2c1ed02c87e77dedd9cb604d2996e60e
Jul 28 22:58:53.467705 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:53.467680    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480650928"
Jul 28 22:58:53.467801 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:53.467726    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17b58cf48528f2c7aadc1d6fa947778467ebdcf3daccce67f545b22205194728"
Jul 28 22:58:53.468997 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:53.468975    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480672784" event={"ID":"9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb","Type":"ContainerStarted","Data":"b61d000d60a3b5f18fc1d94a32ba64ac2c1ed02c87e77dedd9cb604d2996e60e"}
Jul 28 22:58:53.478379 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:58:53.478353    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7413-140606480650928\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" podUID="30dc4cbe-c56e-4c5c-8ae3-2c87033f220c" pod="opendatahub/test-curl-7413-140606480650928"
Jul 28 22:58:54.408459 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:54.408424    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30dc4cbe-c56e-4c5c-8ae3-2c87033f220c" path="/var/lib/kubelet/pods/30dc4cbe-c56e-4c5c-8ae3-2c87033f220c/volumes"
Jul 28 22:58:54.475374 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:54.475346    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb" containerID="7fb2b6d75cae79d66730d6f1c09d59a69df15ebcc6aa9317f0db6a30034d2502" exitCode=0
Jul 28 22:58:54.475532 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:54.475430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480672784" event={"ID":"9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb","Type":"ContainerDied","Data":"7fb2b6d75cae79d66730d6f1c09d59a69df15ebcc6aa9317f0db6a30034d2502"}
Jul 28 22:58:55.612748 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.612724    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:55.631442 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.631396    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7413-140606480672784_9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb/test-curl-7413-140606480672784/0.log"
Jul 28 22:58:55.654390 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.654359    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:55.656368 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.656346    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:55.716661 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.716634    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4rhx\" (UniqueName: \"kubernetes.io/projected/9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb-kube-api-access-n4rhx\") pod \"9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb\" (UID: \"9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb\") "
Jul 28 22:58:55.718693 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.718667    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb-kube-api-access-n4rhx" (OuterVolumeSpecName: "kube-api-access-n4rhx") pod "9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb" (UID: "9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb"). InnerVolumeSpecName "kube-api-access-n4rhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:58:55.781600 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.781530    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:55.781933 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.781920    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb" containerName="test-curl-7413-140606480672784"
Jul 28 22:58:55.781983 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.781934    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb" containerName="test-curl-7413-140606480672784"
Jul 28 22:58:55.782029 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.782020    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb" containerName="test-curl-7413-140606480672784"
Jul 28 22:58:55.785176 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.785160    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:55.790340 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.790313    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:55.817973 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.817951    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4rhx\" (UniqueName: \"kubernetes.io/projected/9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb-kube-api-access-n4rhx\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:58:55.919268 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:55.919234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkszm\" (UniqueName: \"kubernetes.io/projected/f0ff21fc-77fd-442b-ba9b-a106e6674332-kube-api-access-mkszm\") pod \"test-curl-7413-140606480672784\" (UID: \"f0ff21fc-77fd-442b-ba9b-a106e6674332\") " pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:56.019979 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:56.019943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkszm\" (UniqueName: \"kubernetes.io/projected/f0ff21fc-77fd-442b-ba9b-a106e6674332-kube-api-access-mkszm\") pod \"test-curl-7413-140606480672784\" (UID: \"f0ff21fc-77fd-442b-ba9b-a106e6674332\") " pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:56.027683 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:56.027655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkszm\" (UniqueName: \"kubernetes.io/projected/f0ff21fc-77fd-442b-ba9b-a106e6674332-kube-api-access-mkszm\") pod \"test-curl-7413-140606480672784\" (UID: \"f0ff21fc-77fd-442b-ba9b-a106e6674332\") " pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:56.096128 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:56.096040    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:56.229944 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:56.229917    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:56.232555 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:58:56.232529    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0ff21fc_77fd_442b_ba9b_a106e6674332.slice/crio-88372734c9e6ec449295234cca92380ac6fea5872ae6bb109f19949aa828b306 WatchSource:0}: Error finding container 88372734c9e6ec449295234cca92380ac6fea5872ae6bb109f19949aa828b306: Status 404 returned error can't find the container with id 88372734c9e6ec449295234cca92380ac6fea5872ae6bb109f19949aa828b306
Jul 28 22:58:56.408598 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:56.408520    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb" path="/var/lib/kubelet/pods/9f5ae2f5-d5fe-497f-88d9-c65a4fdadccb/volumes"
Jul 28 22:58:56.487155 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:56.487123    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480672784" event={"ID":"f0ff21fc-77fd-442b-ba9b-a106e6674332","Type":"ContainerStarted","Data":"88372734c9e6ec449295234cca92380ac6fea5872ae6bb109f19949aa828b306"}
Jul 28 22:58:56.488350 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:56.488334    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:56.488350 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:56.488349    2560 scope.go:117] "RemoveContainer" containerID="7fb2b6d75cae79d66730d6f1c09d59a69df15ebcc6aa9317f0db6a30034d2502"
Jul 28 22:58:57.495439 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:57.495405    2560 generic.go:356] "Generic (PLEG): container finished" podID="f0ff21fc-77fd-442b-ba9b-a106e6674332" containerID="95b4b40af7316d38d201e3c81ff4377246c7a897b13fa9144af9ad8f47db1c78" exitCode=0
Jul 28 22:58:57.495862 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:57.495486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480672784" event={"ID":"f0ff21fc-77fd-442b-ba9b-a106e6674332","Type":"ContainerDied","Data":"95b4b40af7316d38d201e3c81ff4377246c7a897b13fa9144af9ad8f47db1c78"}
Jul 28 22:58:58.639715 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.639681    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:58.658734 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.658693    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7413-140606480672784_f0ff21fc-77fd-442b-ba9b-a106e6674332/test-curl-7413-140606480672784/0.log"
Jul 28 22:58:58.682435 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.682408    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:58.685864 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.685843    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7413-140606480672784"]
Jul 28 22:58:58.746111 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.746077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkszm\" (UniqueName: \"kubernetes.io/projected/f0ff21fc-77fd-442b-ba9b-a106e6674332-kube-api-access-mkszm\") pod \"f0ff21fc-77fd-442b-ba9b-a106e6674332\" (UID: \"f0ff21fc-77fd-442b-ba9b-a106e6674332\") "
Jul 28 22:58:58.748045 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.748013    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ff21fc-77fd-442b-ba9b-a106e6674332-kube-api-access-mkszm" (OuterVolumeSpecName: "kube-api-access-mkszm") pod "f0ff21fc-77fd-442b-ba9b-a106e6674332" (UID: "f0ff21fc-77fd-442b-ba9b-a106e6674332"). InnerVolumeSpecName "kube-api-access-mkszm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:58:58.810914 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.810884    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:58:58.811508 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.811490    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ff21fc-77fd-442b-ba9b-a106e6674332" containerName="test-curl-7413-140606480672784"
Jul 28 22:58:58.811577 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.811512    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ff21fc-77fd-442b-ba9b-a106e6674332" containerName="test-curl-7413-140606480672784"
Jul 28 22:58:58.811656 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.811644    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ff21fc-77fd-442b-ba9b-a106e6674332" containerName="test-curl-7413-140606480672784"
Jul 28 22:58:58.814972 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.814951    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:58:58.818357 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.818331    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:58:58.847632 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.847608    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkszm\" (UniqueName: \"kubernetes.io/projected/f0ff21fc-77fd-442b-ba9b-a106e6674332-kube-api-access-mkszm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:58:58.948501 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:58.948460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhv4\" (UniqueName: \"kubernetes.io/projected/3f722249-4145-4ac8-b589-fabeaef04d52-kube-api-access-vrhv4\") pod \"test-curl-7413-140606480673792\" (UID: \"3f722249-4145-4ac8-b589-fabeaef04d52\") " pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:58:59.049574 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:59.049486    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhv4\" (UniqueName: \"kubernetes.io/projected/3f722249-4145-4ac8-b589-fabeaef04d52-kube-api-access-vrhv4\") pod \"test-curl-7413-140606480673792\" (UID: \"3f722249-4145-4ac8-b589-fabeaef04d52\") " pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:58:59.057142 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:59.057111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhv4\" (UniqueName: \"kubernetes.io/projected/3f722249-4145-4ac8-b589-fabeaef04d52-kube-api-access-vrhv4\") pod \"test-curl-7413-140606480673792\" (UID: \"3f722249-4145-4ac8-b589-fabeaef04d52\") " pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:58:59.125745 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:59.125717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:58:59.464480 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:59.464431    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:58:59.466760 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:58:59.466734    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f722249_4145_4ac8_b589_fabeaef04d52.slice/crio-5f0feecd846887a197e5658e2fa87467ada3cd4a47683e8adc2314272486f226 WatchSource:0}: Error finding container 5f0feecd846887a197e5658e2fa87467ada3cd4a47683e8adc2314272486f226: Status 404 returned error can't find the container with id 5f0feecd846887a197e5658e2fa87467ada3cd4a47683e8adc2314272486f226
Jul 28 22:58:59.507569 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:59.507547    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:59.507693 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:59.507593    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88372734c9e6ec449295234cca92380ac6fea5872ae6bb109f19949aa828b306"
Jul 28 22:58:59.508837 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:59.508811    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480673792" event={"ID":"3f722249-4145-4ac8-b589-fabeaef04d52","Type":"ContainerStarted","Data":"5f0feecd846887a197e5658e2fa87467ada3cd4a47683e8adc2314272486f226"}
Jul 28 22:58:59.518286 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:58:59.518260    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7413-140606480672784\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" podUID="f0ff21fc-77fd-442b-ba9b-a106e6674332" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:58:59.769391 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:58:59.769352    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4" podUID="dec65986-7061-4fbd-b6ed-87c1d8f03705" containerName="main" containerID="cri-o://3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0" gracePeriod=45
Jul 28 22:59:00.022446 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.022424    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:59:00.039726 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:59:00.039683    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7413-140606480672784\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" podUID="f0ff21fc-77fd-442b-ba9b-a106e6674332" pod="opendatahub/test-curl-7413-140606480672784"
Jul 28 22:59:00.159376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159344    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-tmp-dir\") pod \"dec65986-7061-4fbd-b6ed-87c1d8f03705\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") "
Jul 28 22:59:00.159376 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159383    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec65986-7061-4fbd-b6ed-87c1d8f03705-tls-certs\") pod \"dec65986-7061-4fbd-b6ed-87c1d8f03705\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") "
Jul 28 22:59:00.159599 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159422    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t75x2\" (UniqueName: \"kubernetes.io/projected/dec65986-7061-4fbd-b6ed-87c1d8f03705-kube-api-access-t75x2\") pod \"dec65986-7061-4fbd-b6ed-87c1d8f03705\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") "
Jul 28 22:59:00.159599 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-dshm\") pod \"dec65986-7061-4fbd-b6ed-87c1d8f03705\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") "
Jul 28 22:59:00.159599 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159508    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-home\") pod \"dec65986-7061-4fbd-b6ed-87c1d8f03705\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") "
Jul 28 22:59:00.159599 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159553    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-model-cache\") pod \"dec65986-7061-4fbd-b6ed-87c1d8f03705\" (UID: \"dec65986-7061-4fbd-b6ed-87c1d8f03705\") "
Jul 28 22:59:00.159807 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159623    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dec65986-7061-4fbd-b6ed-87c1d8f03705" (UID: "dec65986-7061-4fbd-b6ed-87c1d8f03705"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:59:00.159866 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159804    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-home" (OuterVolumeSpecName: "home") pod "dec65986-7061-4fbd-b6ed-87c1d8f03705" (UID: "dec65986-7061-4fbd-b6ed-87c1d8f03705"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:59:00.159915 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159887    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-model-cache" (OuterVolumeSpecName: "model-cache") pod "dec65986-7061-4fbd-b6ed-87c1d8f03705" (UID: "dec65986-7061-4fbd-b6ed-87c1d8f03705"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:59:00.160014 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159976    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:59:00.160014 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.159998    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:59:00.161615 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.161589    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-dshm" (OuterVolumeSpecName: "dshm") pod "dec65986-7061-4fbd-b6ed-87c1d8f03705" (UID: "dec65986-7061-4fbd-b6ed-87c1d8f03705"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:59:00.162040 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.162019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec65986-7061-4fbd-b6ed-87c1d8f03705-kube-api-access-t75x2" (OuterVolumeSpecName: "kube-api-access-t75x2") pod "dec65986-7061-4fbd-b6ed-87c1d8f03705" (UID: "dec65986-7061-4fbd-b6ed-87c1d8f03705"). InnerVolumeSpecName "kube-api-access-t75x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:59:00.162133 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.162024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec65986-7061-4fbd-b6ed-87c1d8f03705-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dec65986-7061-4fbd-b6ed-87c1d8f03705" (UID: "dec65986-7061-4fbd-b6ed-87c1d8f03705"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:59:00.260780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.260696    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t75x2\" (UniqueName: \"kubernetes.io/projected/dec65986-7061-4fbd-b6ed-87c1d8f03705-kube-api-access-t75x2\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:59:00.260780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.260723    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:59:00.260780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.260733    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec65986-7061-4fbd-b6ed-87c1d8f03705-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:59:00.260780 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.260741    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec65986-7061-4fbd-b6ed-87c1d8f03705-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:59:00.407945 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.407911    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ff21fc-77fd-442b-ba9b-a106e6674332" path="/var/lib/kubelet/pods/f0ff21fc-77fd-442b-ba9b-a106e6674332/volumes"
Jul 28 22:59:00.515377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.515288    2560 generic.go:356] "Generic (PLEG): container finished" podID="dec65986-7061-4fbd-b6ed-87c1d8f03705" containerID="3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0" exitCode=0
Jul 28 22:59:00.515377 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.515335    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"
Jul 28 22:59:00.515596 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.515371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4" event={"ID":"dec65986-7061-4fbd-b6ed-87c1d8f03705","Type":"ContainerDied","Data":"3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0"}
Jul 28 22:59:00.515596 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.515413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4" event={"ID":"dec65986-7061-4fbd-b6ed-87c1d8f03705","Type":"ContainerDied","Data":"32ccae395c1b8dff7ba8f928b3663a066220024266943262b59694bd7e9b6a2c"}
Jul 28 22:59:00.515596 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.515435    2560 scope.go:117] "RemoveContainer" containerID="3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0"
Jul 28 22:59:00.517180 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.517156    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f722249-4145-4ac8-b589-fabeaef04d52" containerID="1464981486c3706397fc6fd15da26d8572740365acba849f7d777c514b9247bc" exitCode=0
Jul 28 22:59:00.517305 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.517183    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480673792" event={"ID":"3f722249-4145-4ac8-b589-fabeaef04d52","Type":"ContainerDied","Data":"1464981486c3706397fc6fd15da26d8572740365acba849f7d777c514b9247bc"}
Jul 28 22:59:00.524191 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.524172    2560 scope.go:117] "RemoveContainer" containerID="3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0"
Jul 28 22:59:00.524453 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:59:00.524428    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0\": container with ID starting with 3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0 not found: ID does not exist" containerID="3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0"
Jul 28 22:59:00.524505 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.524461    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0"} err="failed to get container status \"3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0\": rpc error: code = NotFound desc = could not find container \"3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0\": container with ID starting with 3ff051bb8221d0523998de9cc98145f051da0629fa8cac6463301e21083e26d0 not found: ID does not exist"
Jul 28 22:59:00.543671 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.543644    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"]
Jul 28 22:59:00.545545 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:00.545521    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a134df/rate-test-model-a134df-kserve-5479dcccd7-g4cg4"]
Jul 28 22:59:01.660149 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.660123    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:01.680465 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.680434    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7413-140606480673792_3f722249-4145-4ac8-b589-fabeaef04d52/test-curl-7413-140606480673792/0.log"
Jul 28 22:59:01.706968 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.706934    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:59:01.712727 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.712704    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:59:01.775796 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.775765    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrhv4\" (UniqueName: \"kubernetes.io/projected/3f722249-4145-4ac8-b589-fabeaef04d52-kube-api-access-vrhv4\") pod \"3f722249-4145-4ac8-b589-fabeaef04d52\" (UID: \"3f722249-4145-4ac8-b589-fabeaef04d52\") "
Jul 28 22:59:01.777852 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.777794    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f722249-4145-4ac8-b589-fabeaef04d52-kube-api-access-vrhv4" (OuterVolumeSpecName: "kube-api-access-vrhv4") pod "3f722249-4145-4ac8-b589-fabeaef04d52" (UID: "3f722249-4145-4ac8-b589-fabeaef04d52"). InnerVolumeSpecName "kube-api-access-vrhv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:59:01.840445 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.840412    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:59:01.840812 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.840800    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec65986-7061-4fbd-b6ed-87c1d8f03705" containerName="main"
Jul 28 22:59:01.840857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.840813    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec65986-7061-4fbd-b6ed-87c1d8f03705" containerName="main"
Jul 28 22:59:01.840857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.840839    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f722249-4145-4ac8-b589-fabeaef04d52" containerName="test-curl-7413-140606480673792"
Jul 28 22:59:01.840857 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.840847    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f722249-4145-4ac8-b589-fabeaef04d52" containerName="test-curl-7413-140606480673792"
Jul 28 22:59:01.840948 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.840907    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f722249-4145-4ac8-b589-fabeaef04d52" containerName="test-curl-7413-140606480673792"
Jul 28 22:59:01.840948 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.840917    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dec65986-7061-4fbd-b6ed-87c1d8f03705" containerName="main"
Jul 28 22:59:01.850575 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.850543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:01.850776 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.850751    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:59:01.877292 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.877265    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrhv4\" (UniqueName: \"kubernetes.io/projected/3f722249-4145-4ac8-b589-fabeaef04d52-kube-api-access-vrhv4\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:59:01.978160 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:01.978125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfw4h\" (UniqueName: \"kubernetes.io/projected/78009567-5732-420e-b40d-0e9e2134cf58-kube-api-access-tfw4h\") pod \"test-curl-7413-140606480673792\" (UID: \"78009567-5732-420e-b40d-0e9e2134cf58\") " pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:02.079524 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:02.079441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfw4h\" (UniqueName: \"kubernetes.io/projected/78009567-5732-420e-b40d-0e9e2134cf58-kube-api-access-tfw4h\") pod \"test-curl-7413-140606480673792\" (UID: \"78009567-5732-420e-b40d-0e9e2134cf58\") " pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:02.088149 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:02.088093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfw4h\" (UniqueName: \"kubernetes.io/projected/78009567-5732-420e-b40d-0e9e2134cf58-kube-api-access-tfw4h\") pod \"test-curl-7413-140606480673792\" (UID: \"78009567-5732-420e-b40d-0e9e2134cf58\") " pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:02.160639 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:02.160582    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:02.291039 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:02.291016    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:59:02.293145 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:59:02.293094    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78009567_5732_420e_b40d_0e9e2134cf58.slice/crio-82a712db531ac53496234a2b6f21cf70b08a902a605c9306e5dec7938044b82d WatchSource:0}: Error finding container 82a712db531ac53496234a2b6f21cf70b08a902a605c9306e5dec7938044b82d: Status 404 returned error can't find the container with id 82a712db531ac53496234a2b6f21cf70b08a902a605c9306e5dec7938044b82d
Jul 28 22:59:02.407065 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:02.407028    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f722249-4145-4ac8-b589-fabeaef04d52" path="/var/lib/kubelet/pods/3f722249-4145-4ac8-b589-fabeaef04d52/volumes"
Jul 28 22:59:02.407488 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:02.407473    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dec65986-7061-4fbd-b6ed-87c1d8f03705" path="/var/lib/kubelet/pods/dec65986-7061-4fbd-b6ed-87c1d8f03705/volumes"
Jul 28 22:59:02.533080 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:02.533052    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:02.533292 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:02.533065    2560 scope.go:117] "RemoveContainer" containerID="1464981486c3706397fc6fd15da26d8572740365acba849f7d777c514b9247bc"
Jul 28 22:59:02.534450 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:02.534290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480673792" event={"ID":"78009567-5732-420e-b40d-0e9e2134cf58","Type":"ContainerStarted","Data":"82a712db531ac53496234a2b6f21cf70b08a902a605c9306e5dec7938044b82d"}
Jul 28 22:59:03.541603 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:03.541570    2560 generic.go:356] "Generic (PLEG): container finished" podID="78009567-5732-420e-b40d-0e9e2134cf58" containerID="916d23822473c9d01a1e038d81a5abee4d4d2b1a2576eb374605d32cdabec98b" exitCode=0
Jul 28 22:59:03.542007 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:03.541653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7413-140606480673792" event={"ID":"78009567-5732-420e-b40d-0e9e2134cf58","Type":"ContainerDied","Data":"916d23822473c9d01a1e038d81a5abee4d4d2b1a2576eb374605d32cdabec98b"}
Jul 28 22:59:04.688036 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:04.688008    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:04.704399 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:04.704365    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfw4h\" (UniqueName: \"kubernetes.io/projected/78009567-5732-420e-b40d-0e9e2134cf58-kube-api-access-tfw4h\") pod \"78009567-5732-420e-b40d-0e9e2134cf58\" (UID: \"78009567-5732-420e-b40d-0e9e2134cf58\") "
Jul 28 22:59:04.706607 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:04.706576    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78009567-5732-420e-b40d-0e9e2134cf58-kube-api-access-tfw4h" (OuterVolumeSpecName: "kube-api-access-tfw4h") pod "78009567-5732-420e-b40d-0e9e2134cf58" (UID: "78009567-5732-420e-b40d-0e9e2134cf58"). InnerVolumeSpecName "kube-api-access-tfw4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:59:04.707361 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:04.707334    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7413-140606480673792_78009567-5732-420e-b40d-0e9e2134cf58/test-curl-7413-140606480673792/0.log"
Jul 28 22:59:04.730702 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:04.730672    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:59:04.734569 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:04.734547    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7413-140606480673792"]
Jul 28 22:59:04.805816 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:04.805737    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfw4h\" (UniqueName: \"kubernetes.io/projected/78009567-5732-420e-b40d-0e9e2134cf58-kube-api-access-tfw4h\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 22:59:05.231216 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:05.231181    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:05.554528 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:05.554437    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a712db531ac53496234a2b6f21cf70b08a902a605c9306e5dec7938044b82d"
Jul 28 22:59:05.554528 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:05.554458    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:05.565549 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:59:05.565518    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7413-140606480673792\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" podUID="78009567-5732-420e-b40d-0e9e2134cf58" pod="opendatahub/test-curl-7413-140606480673792"
Jul 28 22:59:05.622230 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:05.622197    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:05.738093 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:05.738051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:06.407781 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:06.407752    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78009567-5732-420e-b40d-0e9e2134cf58" path="/var/lib/kubelet/pods/78009567-5732-420e-b40d-0e9e2134cf58/volumes"
Jul 28 22:59:09.131288 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.131244    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:09.211230 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.211191    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:09.277236 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.277186    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:09.689829 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.689790    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:09.817966 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.817936    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:09.908358 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.908318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:09.952373 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.952302    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"]
Jul 28 22:59:09.952739 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.952725    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78009567-5732-420e-b40d-0e9e2134cf58" containerName="test-curl-7413-140606480673792"
Jul 28 22:59:09.952779 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.952740    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78009567-5732-420e-b40d-0e9e2134cf58" containerName="test-curl-7413-140606480673792"
Jul 28 22:59:09.952810 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.952805    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78009567-5732-420e-b40d-0e9e2134cf58" containerName="test-curl-7413-140606480673792"
Jul 28 22:59:09.955796 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.955778    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:09.958126 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.958091    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c6c731\""
Jul 28 22:59:09.958241 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.958131    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c6c731-dockercfg-pq7zp\""
Jul 28 22:59:09.965129 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:09.965094    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"]
Jul 28 22:59:10.059481 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:10.059448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rprc\" (UniqueName: \"kubernetes.io/projected/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-kube-api-access-8rprc\") pod \"payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v\" (UID: \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:10.059625 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:10.059515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v\" (UID: \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:10.160203 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:10.160168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rprc\" (UniqueName: \"kubernetes.io/projected/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-kube-api-access-8rprc\") pod \"payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v\" (UID: \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:10.160570 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:10.160231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v\" (UID: \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:10.160867 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:10.160848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v\" (UID: \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:10.187214 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:10.187190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rprc\" (UniqueName: \"kubernetes.io/projected/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-kube-api-access-8rprc\") pod \"payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v\" (UID: \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:10.267468 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:10.267440    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:10.609351 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:10.609328    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"]
Jul 28 22:59:10.611395 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:59:10.611369    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7056e9_ea6f_4eff_861d_87d4a41f5e7f.slice/crio-ea47789e562355ee83cd936de310de1ecdbfc52414743bda76af8b165e85d8cf WatchSource:0}: Error finding container ea47789e562355ee83cd936de310de1ecdbfc52414743bda76af8b165e85d8cf: Status 404 returned error can't find the container with id ea47789e562355ee83cd936de310de1ecdbfc52414743bda76af8b165e85d8cf
Jul 28 22:59:11.590547 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:11.590512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v" event={"ID":"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f","Type":"ContainerStarted","Data":"01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026"}
Jul 28 22:59:11.590944 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:11.590565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v" event={"ID":"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f","Type":"ContainerStarted","Data":"ea47789e562355ee83cd936de310de1ecdbfc52414743bda76af8b165e85d8cf"}
Jul 28 22:59:11.590944 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:11.590624    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:11.607778 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:11.607732    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v" podStartSLOduration=2.60771727 podStartE2EDuration="2.60771727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:59:11.606041178 +0000 UTC m=+2953.728389073" watchObservedRunningTime="2026-07-28 22:59:11.60771727 +0000 UTC m=+2953.730065191"
Jul 28 22:59:22.599009 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:22.598940    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 22:59:34.059086 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:34.059046    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:34.751308 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:34.751272    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:35.093487 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:35.093392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:35.359193 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:35.359081    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:38.241649 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.241618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:38.346358 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.346318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:38.811686 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.811647    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"]
Jul 28 22:59:38.815316 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.815298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:38.817603 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.817582    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6fb213-serving-cert\""
Jul 28 22:59:38.823631 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.823605    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"]
Jul 28 22:59:38.832583 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.832558    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:38.941946 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.941908    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:38.942160 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.942042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:38.942160 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:38.942140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqpg\" (UniqueName: \"kubernetes.io/projected/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-kube-api-access-6gqpg\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:39.043226 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.043179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:39.043428 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.043251    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqpg\" (UniqueName: \"kubernetes.io/projected/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-kube-api-access-6gqpg\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:39.043428 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.043296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:39.043428 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:59:39.043331    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-serving-cert: secret "maas-api-e2e-ipp-b-6fb213-serving-cert" not found
Jul 28 22:59:39.043428 ip-10-0-129-50 kubenswrapper[2560]: E0728 22:59:39.043402    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-maas-api-tls podName:87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe nodeName:}" failed. No retries permitted until 2026-07-28 22:59:39.543384909 +0000 UTC m=+2981.665732782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-maas-api-tls") pod "maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd" (UID: "87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe") : secret "maas-api-e2e-ipp-b-6fb213-serving-cert" not found
Jul 28 22:59:39.043885 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.043867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:39.052286 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.052259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqpg\" (UniqueName: \"kubernetes.io/projected/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-kube-api-access-6gqpg\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:39.102249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.102185    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:39.298498 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.298461    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:39.399621 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.399531    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:39.408775 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.408743    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"]
Jul 28 22:59:39.412393 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.412377    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:39.415078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.415031    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6fb213-dockercfg-96wlm\""
Jul 28 22:59:39.415078 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.415048    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6fb213\""
Jul 28 22:59:39.421398 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.421378    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"]
Jul 28 22:59:39.426028 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.426007    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"]
Jul 28 22:59:39.429369 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.429352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:39.441000 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.440978    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"]
Jul 28 22:59:39.547649 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.547615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm5vw\" (UniqueName: \"kubernetes.io/projected/20a41c81-022c-45f3-b193-5ee09aecbf08-kube-api-access-gm5vw\") pod \"payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp\" (UID: \"20a41c81-022c-45f3-b193-5ee09aecbf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:39.547855 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.547660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:39.547855 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.547682    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f02181-8930-4262-8a8b-d5df9c2936f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v\" (UID: \"44f02181-8930-4262-8a8b-d5df9c2936f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:39.547855 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.547732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a41c81-022c-45f3-b193-5ee09aecbf08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp\" (UID: \"20a41c81-022c-45f3-b193-5ee09aecbf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:39.547855 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.547781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9sq\" (UniqueName: \"kubernetes.io/projected/44f02181-8930-4262-8a8b-d5df9c2936f1-kube-api-access-jg9sq\") pod \"payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v\" (UID: \"44f02181-8930-4262-8a8b-d5df9c2936f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:39.550077 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.550051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:39.649249 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.649203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a41c81-022c-45f3-b193-5ee09aecbf08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp\" (UID: \"20a41c81-022c-45f3-b193-5ee09aecbf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:39.649451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.649293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9sq\" (UniqueName: \"kubernetes.io/projected/44f02181-8930-4262-8a8b-d5df9c2936f1-kube-api-access-jg9sq\") pod \"payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v\" (UID: \"44f02181-8930-4262-8a8b-d5df9c2936f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:39.649451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.649386    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm5vw\" (UniqueName: \"kubernetes.io/projected/20a41c81-022c-45f3-b193-5ee09aecbf08-kube-api-access-gm5vw\") pod \"payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp\" (UID: \"20a41c81-022c-45f3-b193-5ee09aecbf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:39.649451 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.649416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f02181-8930-4262-8a8b-d5df9c2936f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v\" (UID: \"44f02181-8930-4262-8a8b-d5df9c2936f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:39.650035 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.649977    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a41c81-022c-45f3-b193-5ee09aecbf08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp\" (UID: \"20a41c81-022c-45f3-b193-5ee09aecbf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:39.650147 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.650131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f02181-8930-4262-8a8b-d5df9c2936f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v\" (UID: \"44f02181-8930-4262-8a8b-d5df9c2936f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:39.659023 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.659000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm5vw\" (UniqueName: \"kubernetes.io/projected/20a41c81-022c-45f3-b193-5ee09aecbf08-kube-api-access-gm5vw\") pod \"payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp\" (UID: \"20a41c81-022c-45f3-b193-5ee09aecbf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:39.659145 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.659124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9sq\" (UniqueName: \"kubernetes.io/projected/44f02181-8930-4262-8a8b-d5df9c2936f1-kube-api-access-jg9sq\") pod \"payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v\" (UID: \"44f02181-8930-4262-8a8b-d5df9c2936f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:39.724410 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.724377    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:39.727468 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.727447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:39.741497 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:39.741465    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:40.083037 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.083011    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"]
Jul 28 22:59:40.084760 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:59:40.084733    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f02181_8930_4262_8a8b_d5df9c2936f1.slice/crio-1976d0eea57b98f8590498a7cd40b67da94e3b032e073b043938a19660f2f4d6 WatchSource:0}: Error finding container 1976d0eea57b98f8590498a7cd40b67da94e3b032e073b043938a19660f2f4d6: Status 404 returned error can't find the container with id 1976d0eea57b98f8590498a7cd40b67da94e3b032e073b043938a19660f2f4d6
Jul 28 22:59:40.116327 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.116304    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"]
Jul 28 22:59:40.119442 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.119423    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"]
Jul 28 22:59:40.121313 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:59:40.121277    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a41c81_022c_45f3_b193_5ee09aecbf08.slice/crio-b89bb73f38689b2f004667dde69ca6b47ba4b8d2a9d4f4c5bd914c8bb685cc72 WatchSource:0}: Error finding container b89bb73f38689b2f004667dde69ca6b47ba4b8d2a9d4f4c5bd914c8bb685cc72: Status 404 returned error can't find the container with id b89bb73f38689b2f004667dde69ca6b47ba4b8d2a9d4f4c5bd914c8bb685cc72
Jul 28 22:59:40.121827 ip-10-0-129-50 kubenswrapper[2560]: W0728 22:59:40.121802    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c6c0bf_a84e_4dda_b5da_74e5ff8ee9fe.slice/crio-7c4941aac1c7862db156ce19b08c220e89ad0c51bc97a35ab9b58e60350a9240 WatchSource:0}: Error finding container 7c4941aac1c7862db156ce19b08c220e89ad0c51bc97a35ab9b58e60350a9240: Status 404 returned error can't find the container with id 7c4941aac1c7862db156ce19b08c220e89ad0c51bc97a35ab9b58e60350a9240
Jul 28 22:59:40.749197 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.749157    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v" event={"ID":"44f02181-8930-4262-8a8b-d5df9c2936f1","Type":"ContainerStarted","Data":"84b468a332acdf8b19d5ac2a63a1ec902fd118e2f2a00d276ceabcc726b6fc5e"}
Jul 28 22:59:40.749664 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.749228    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:40.749664 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.749253    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v" event={"ID":"44f02181-8930-4262-8a8b-d5df9c2936f1","Type":"ContainerStarted","Data":"1976d0eea57b98f8590498a7cd40b67da94e3b032e073b043938a19660f2f4d6"}
Jul 28 22:59:40.750784 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.750759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd" event={"ID":"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe","Type":"ContainerStarted","Data":"7b77cf7dbc21327cdeecf5e7ba2413181ed85d70d1737934f71ef94b2b5d59c9"}
Jul 28 22:59:40.750930 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.750805    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd" event={"ID":"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe","Type":"ContainerStarted","Data":"7c4941aac1c7862db156ce19b08c220e89ad0c51bc97a35ab9b58e60350a9240"}
Jul 28 22:59:40.750930 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.750833    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:40.752208 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.752184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp" event={"ID":"20a41c81-022c-45f3-b193-5ee09aecbf08","Type":"ContainerStarted","Data":"5768f977ea49b1b240b51e852efc0360aeb12155abd6cec7edc986533a6f3dda"}
Jul 28 22:59:40.752330 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.752225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp" event={"ID":"20a41c81-022c-45f3-b193-5ee09aecbf08","Type":"ContainerStarted","Data":"b89bb73f38689b2f004667dde69ca6b47ba4b8d2a9d4f4c5bd914c8bb685cc72"}
Jul 28 22:59:40.752330 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.752244    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:40.764707 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.764662    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v" podStartSLOduration=1.764647111 podStartE2EDuration="1.764647111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:59:40.763801467 +0000 UTC m=+2982.886149376" watchObservedRunningTime="2026-07-28 22:59:40.764647111 +0000 UTC m=+2982.886995003"
Jul 28 22:59:40.785571 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.785525    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd" podStartSLOduration=2.456232594 podStartE2EDuration="2.785508377s" totalImagesPullingTime="329.275783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:59:40.78347437 +0000 UTC m=+2982.905822316" watchObservedRunningTime="2026-07-28 22:59:40.785508377 +0000 UTC m=+2982.907856269"
Jul 28 22:59:40.800211 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:40.800169    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp" podStartSLOduration=1.8001539960000001 podStartE2EDuration="1.800153996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:59:40.798886828 +0000 UTC m=+2982.921234722" watchObservedRunningTime="2026-07-28 22:59:40.800153996 +0000 UTC m=+2982.922501887"
Jul 28 22:59:46.763094 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:46.763063    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 22:59:51.760305 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:51.760271    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 22:59:51.760828 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:51.760529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 22:59:52.897832 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:52.897792    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 22:59:59.778862 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:59.778826    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7g7wm_3e34ca30-b28c-4cca-8472-d23bcf1ddf51/payload-processing/0.log"
Jul 28 22:59:59.893795 ip-10-0-129-50 kubenswrapper[2560]: I0728 22:59:59.893764    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp_20a41c81-022c-45f3-b193-5ee09aecbf08/payload-processing/0.log"
Jul 28 23:00:00.134653 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.134569    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb"]
Jul 28 23:00:00.139699 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.139677    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5"]
Jul 28 23:00:00.139812 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.139785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb"
Jul 28 23:00:00.142161 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.142137    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 23:00:00.143540 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.143479    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql"]
Jul 28 23:00:00.143746 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.143728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5"
Jul 28 23:00:00.147760 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.147738    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn"]
Jul 28 23:00:00.147875 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.147847    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql"
Jul 28 23:00:00.152003 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.151966    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk"]
Jul 28 23:00:00.152347 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.152328    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn"
Jul 28 23:00:00.155601 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.155577    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb"
Jul 28 23:00:00.158854 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.158761    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb"]
Jul 28 23:00:00.158854 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.158791    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5"]
Jul 28 23:00:00.158854 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.158809    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql"]
Jul 28 23:00:00.158854 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.158827    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn"]
Jul 28 23:00:00.159066 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.158897    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk"
Jul 28 23:00:00.165841 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.165820    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk"]
Jul 28 23:00:00.166340 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.166323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5"
Jul 28 23:00:00.207923 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.207898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql"
Jul 28 23:00:00.218972 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.218947    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn"
Jul 28 23:00:00.262806 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.262780    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk"
Jul 28 23:00:00.332660 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.328753    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb"]
Jul 28 23:00:00.332781 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:00:00.331933    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4db883e_ad1c_4449_97cd_c8fda4f50eb7.slice/crio-8388facb3b680d6bdb1e49b025dfdf08dfb5c6a3c2abeeef1999225f830e7ac9 WatchSource:0}: Error finding container 8388facb3b680d6bdb1e49b025dfdf08dfb5c6a3c2abeeef1999225f830e7ac9: Status 404 returned error can't find the container with id 8388facb3b680d6bdb1e49b025dfdf08dfb5c6a3c2abeeef1999225f830e7ac9
Jul 28 23:00:00.385220 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.385154    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5"]
Jul 28 23:00:00.387727 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:00:00.387285    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d8eab9_7eed_4cf3_8ca4_1ea57b2b36ee.slice/crio-11d0806fcc61f54fa122e6cc61cc8bac0d74cb017905e37d6f824e6fc7a79828 WatchSource:0}: Error finding container 11d0806fcc61f54fa122e6cc61cc8bac0d74cb017905e37d6f824e6fc7a79828: Status 404 returned error can't find the container with id 11d0806fcc61f54fa122e6cc61cc8bac0d74cb017905e37d6f824e6fc7a79828
Jul 28 23:00:00.413149 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.413110    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql"]
Jul 28 23:00:00.414972 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:00:00.414928    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c78c7da_bb55_4c16_a609_130a158d0228.slice/crio-71941d57866da1d90c4018b4bb034ad2f1f0bed30b428225e4624213ec74f585 WatchSource:0}: Error finding container 71941d57866da1d90c4018b4bb034ad2f1f0bed30b428225e4624213ec74f585: Status 404 returned error can't find the container with id 71941d57866da1d90c4018b4bb034ad2f1f0bed30b428225e4624213ec74f585
Jul 28 23:00:00.448391 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.448366    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn"]
Jul 28 23:00:00.459683 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:00:00.459659    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af6d0e1_cf7f_464e_9e97_e1b4529eee82.slice/crio-cd3838315026d6296d9f48af40550bc0159b3fcddd980368494d62076889bf51 WatchSource:0}: Error finding container cd3838315026d6296d9f48af40550bc0159b3fcddd980368494d62076889bf51: Status 404 returned error can't find the container with id cd3838315026d6296d9f48af40550bc0159b3fcddd980368494d62076889bf51
Jul 28 23:00:00.479426 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.479402    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk"]
Jul 28 23:00:00.481092 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:00:00.481068    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b5963a_f1ea_43c9_8082_442f485e958f.slice/crio-1429ef2ac7d1e94c8bb482d856175f988489a57ed5757f576c9aa4e678a24d6a WatchSource:0}: Error finding container 1429ef2ac7d1e94c8bb482d856175f988489a57ed5757f576c9aa4e678a24d6a: Status 404 returned error can't find the container with id 1429ef2ac7d1e94c8bb482d856175f988489a57ed5757f576c9aa4e678a24d6a
Jul 28 23:00:00.861661 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.861630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk" event={"ID":"11b5963a-f1ea-43c9-8082-442f485e958f","Type":"ContainerStarted","Data":"1429ef2ac7d1e94c8bb482d856175f988489a57ed5757f576c9aa4e678a24d6a"}
Jul 28 23:00:00.863052 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.863024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5" event={"ID":"19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee","Type":"ContainerStarted","Data":"f788d46fb3539105c4509109e7da4c0af2353ae62ff2b94b8c7fabf9d851b482"}
Jul 28 23:00:00.863198 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.863079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5" event={"ID":"19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee","Type":"ContainerStarted","Data":"11d0806fcc61f54fa122e6cc61cc8bac0d74cb017905e37d6f824e6fc7a79828"}
Jul 28 23:00:00.864534 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.864516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn" event={"ID":"3af6d0e1-cf7f-464e-9e97-e1b4529eee82","Type":"ContainerStarted","Data":"432e553b5f384c7fcc165b9fb8477b8638887d8efac3b351b994cc3f3b973116"}
Jul 28 23:00:00.864592 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.864558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn" event={"ID":"3af6d0e1-cf7f-464e-9e97-e1b4529eee82","Type":"ContainerStarted","Data":"cd3838315026d6296d9f48af40550bc0159b3fcddd980368494d62076889bf51"}
Jul 28 23:00:00.865949 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.865926    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql" event={"ID":"4c78c7da-bb55-4c16-a609-130a158d0228","Type":"ContainerStarted","Data":"e385d97c56750644350b9712af795e18f65f50ba92e74fae4c86f9008350403c"}
Jul 28 23:00:00.866067 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.865970    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql" event={"ID":"4c78c7da-bb55-4c16-a609-130a158d0228","Type":"ContainerStarted","Data":"71941d57866da1d90c4018b4bb034ad2f1f0bed30b428225e4624213ec74f585"}
Jul 28 23:00:00.867377 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.867357    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb" event={"ID":"b4db883e-ad1c-4449-97cd-c8fda4f50eb7","Type":"ContainerStarted","Data":"a7dbe79937b173144bbaf65ca95e1a8b4290ce9733623ec7bae8c9192c43b6d0"}
Jul 28 23:00:00.867455 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.867392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb" event={"ID":"b4db883e-ad1c-4449-97cd-c8fda4f50eb7","Type":"ContainerStarted","Data":"8388facb3b680d6bdb1e49b025dfdf08dfb5c6a3c2abeeef1999225f830e7ac9"}
Jul 28 23:00:00.879911 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.879875    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5" podStartSLOduration=0.879863661 podStartE2EDuration="879.863661ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:00.877874711 +0000 UTC m=+3003.000222603" watchObservedRunningTime="2026-07-28 23:00:00.879863661 +0000 UTC m=+3003.002211608"
Jul 28 23:00:00.893718 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.893679    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql" podStartSLOduration=0.893666395 podStartE2EDuration="893.666395ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:00.891516327 +0000 UTC m=+3003.013864218" watchObservedRunningTime="2026-07-28 23:00:00.893666395 +0000 UTC m=+3003.016014325"
Jul 28 23:00:00.906196 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.906086    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn" podStartSLOduration=0.906072173 podStartE2EDuration="906.072173ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:00.905249663 +0000 UTC m=+3003.027597555" watchObservedRunningTime="2026-07-28 23:00:00.906072173 +0000 UTC m=+3003.028420115"
Jul 28 23:00:00.924515 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:00.924458    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb" podStartSLOduration=0.924439007 podStartE2EDuration="924.439007ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:00.921353479 +0000 UTC m=+3003.043701372" watchObservedRunningTime="2026-07-28 23:00:00.924439007 +0000 UTC m=+3003.046786898"
Jul 28 23:00:01.874614 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.874522    2560 generic.go:356] "Generic (PLEG): container finished" podID="3af6d0e1-cf7f-464e-9e97-e1b4529eee82" containerID="432e553b5f384c7fcc165b9fb8477b8638887d8efac3b351b994cc3f3b973116" exitCode=0
Jul 28 23:00:01.875014 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.874610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn" event={"ID":"3af6d0e1-cf7f-464e-9e97-e1b4529eee82","Type":"ContainerDied","Data":"432e553b5f384c7fcc165b9fb8477b8638887d8efac3b351b994cc3f3b973116"}
Jul 28 23:00:01.876063 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.876036    2560 generic.go:356] "Generic (PLEG): container finished" podID="4c78c7da-bb55-4c16-a609-130a158d0228" containerID="e385d97c56750644350b9712af795e18f65f50ba92e74fae4c86f9008350403c" exitCode=0
Jul 28 23:00:01.876186 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.876062    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql" event={"ID":"4c78c7da-bb55-4c16-a609-130a158d0228","Type":"ContainerDied","Data":"e385d97c56750644350b9712af795e18f65f50ba92e74fae4c86f9008350403c"}
Jul 28 23:00:01.877503 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.877476    2560 generic.go:356] "Generic (PLEG): container finished" podID="b4db883e-ad1c-4449-97cd-c8fda4f50eb7" containerID="a7dbe79937b173144bbaf65ca95e1a8b4290ce9733623ec7bae8c9192c43b6d0" exitCode=0
Jul 28 23:00:01.877567 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.877504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb" event={"ID":"b4db883e-ad1c-4449-97cd-c8fda4f50eb7","Type":"ContainerDied","Data":"a7dbe79937b173144bbaf65ca95e1a8b4290ce9733623ec7bae8c9192c43b6d0"}
Jul 28 23:00:01.878874 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.878850    2560 generic.go:356] "Generic (PLEG): container finished" podID="11b5963a-f1ea-43c9-8082-442f485e958f" containerID="35040e326ad3b6b507ef02f0e337273e7e81eb0939f22ba6f3be873442dcee0c" exitCode=0
Jul 28 23:00:01.878957 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.878923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk" event={"ID":"11b5963a-f1ea-43c9-8082-442f485e958f","Type":"ContainerDied","Data":"35040e326ad3b6b507ef02f0e337273e7e81eb0939f22ba6f3be873442dcee0c"}
Jul 28 23:00:01.880305 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.880289    2560 generic.go:356] "Generic (PLEG): container finished" podID="19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee" containerID="f788d46fb3539105c4509109e7da4c0af2353ae62ff2b94b8c7fabf9d851b482" exitCode=0
Jul 28 23:00:01.880381 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:01.880309    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5" event={"ID":"19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee","Type":"ContainerDied","Data":"f788d46fb3539105c4509109e7da4c0af2353ae62ff2b94b8c7fabf9d851b482"}
Jul 28 23:00:03.084494 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.082359    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn"
Jul 28 23:00:03.117540 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.117513    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql"
Jul 28 23:00:03.121673 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.121650    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk"
Jul 28 23:00:03.125682 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.125663    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb"
Jul 28 23:00:03.129865 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.129844    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5"
Jul 28 23:00:03.892778 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.892749    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk"
Jul 28 23:00:03.892778 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.892774    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk" event={"ID":"11b5963a-f1ea-43c9-8082-442f485e958f","Type":"ContainerDied","Data":"1429ef2ac7d1e94c8bb482d856175f988489a57ed5757f576c9aa4e678a24d6a"}
Jul 28 23:00:03.893016 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.892825    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1429ef2ac7d1e94c8bb482d856175f988489a57ed5757f576c9aa4e678a24d6a"
Jul 28 23:00:03.894139 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.894116    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5"
Jul 28 23:00:03.894139 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.894132    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5" event={"ID":"19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee","Type":"ContainerDied","Data":"11d0806fcc61f54fa122e6cc61cc8bac0d74cb017905e37d6f824e6fc7a79828"}
Jul 28 23:00:03.894299 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.894169    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11d0806fcc61f54fa122e6cc61cc8bac0d74cb017905e37d6f824e6fc7a79828"
Jul 28 23:00:03.895495 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.895474    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn"
Jul 28 23:00:03.895610 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.895495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn" event={"ID":"3af6d0e1-cf7f-464e-9e97-e1b4529eee82","Type":"ContainerDied","Data":"cd3838315026d6296d9f48af40550bc0159b3fcddd980368494d62076889bf51"}
Jul 28 23:00:03.895610 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.895531    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd3838315026d6296d9f48af40550bc0159b3fcddd980368494d62076889bf51"
Jul 28 23:00:03.896788 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.896771    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql"
Jul 28 23:00:03.896875 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.896798    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql" event={"ID":"4c78c7da-bb55-4c16-a609-130a158d0228","Type":"ContainerDied","Data":"71941d57866da1d90c4018b4bb034ad2f1f0bed30b428225e4624213ec74f585"}
Jul 28 23:00:03.896875 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.896835    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71941d57866da1d90c4018b4bb034ad2f1f0bed30b428225e4624213ec74f585"
Jul 28 23:00:03.898252 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.898227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb" event={"ID":"b4db883e-ad1c-4449-97cd-c8fda4f50eb7","Type":"ContainerDied","Data":"8388facb3b680d6bdb1e49b025dfdf08dfb5c6a3c2abeeef1999225f830e7ac9"}
Jul 28 23:00:03.898342 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.898264    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754660-ph6qb"
Jul 28 23:00:03.898342 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:03.898269    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8388facb3b680d6bdb1e49b025dfdf08dfb5c6a3c2abeeef1999225f830e7ac9"
Jul 28 23:00:17.465656 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.465613    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"]
Jul 28 23:00:17.466198 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466181    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee" containerName="cleanup"
Jul 28 23:00:17.466254 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466201    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee" containerName="cleanup"
Jul 28 23:00:17.466254 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466212    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b5963a-f1ea-43c9-8082-442f485e958f" containerName="cleanup"
Jul 28 23:00:17.466254 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466218    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b5963a-f1ea-43c9-8082-442f485e958f" containerName="cleanup"
Jul 28 23:00:17.466254 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466238    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4db883e-ad1c-4449-97cd-c8fda4f50eb7" containerName="cleanup"
Jul 28 23:00:17.466254 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466244    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4db883e-ad1c-4449-97cd-c8fda4f50eb7" containerName="cleanup"
Jul 28 23:00:17.466254 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466250    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af6d0e1-cf7f-464e-9e97-e1b4529eee82" containerName="cleanup"
Jul 28 23:00:17.466254 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466255    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af6d0e1-cf7f-464e-9e97-e1b4529eee82" containerName="cleanup"
Jul 28 23:00:17.466468 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466260    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c78c7da-bb55-4c16-a609-130a158d0228" containerName="cleanup"
Jul 28 23:00:17.466468 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466265    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c78c7da-bb55-4c16-a609-130a158d0228" containerName="cleanup"
Jul 28 23:00:17.466468 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466328    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee" containerName="cleanup"
Jul 28 23:00:17.466468 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466333    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4db883e-ad1c-4449-97cd-c8fda4f50eb7" containerName="cleanup"
Jul 28 23:00:17.466468 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466341    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c78c7da-bb55-4c16-a609-130a158d0228" containerName="cleanup"
Jul 28 23:00:17.466468 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466351    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af6d0e1-cf7f-464e-9e97-e1b4529eee82" containerName="cleanup"
Jul 28 23:00:17.466468 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.466359    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="11b5963a-f1ea-43c9-8082-442f485e958f" containerName="cleanup"
Jul 28 23:00:17.469565 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.469545    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.473307 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.473280    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c6c731\"/\"default-dockercfg-nbs2m\""
Jul 28 23:00:17.473439 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.473324    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c6c731\"/\"kube-root-ca.crt\""
Jul 28 23:00:17.473439 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.473282    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c6c731\"/\"ipp-route-c6c731-kserve-self-signed-certs\""
Jul 28 23:00:17.473439 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.473293    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c6c731\"/\"openshift-service-ca.crt\""
Jul 28 23:00:17.478119 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.478073    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"]
Jul 28 23:00:17.557640 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.557601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:00:17.614622 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.614589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-dshm\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.614622 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.614624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-home\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.614855 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.614643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tls-certs\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.614855 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.614793    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzcp\" (UniqueName: \"kubernetes.io/projected/4b06a44e-26f8-49e4-99fb-cbd93efbba13-kube-api-access-4lzcp\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.614927 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.614864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-model-cache\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.614927 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.614915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tmp-dir\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.716569 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.716466    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-dshm\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.716569 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.716518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-home\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.716569 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.716540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tls-certs\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.716863 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.716598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzcp\" (UniqueName: \"kubernetes.io/projected/4b06a44e-26f8-49e4-99fb-cbd93efbba13-kube-api-access-4lzcp\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.716863 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.716651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-model-cache\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.716863 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.716696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tmp-dir\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.717019 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.716960    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-home\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.717019 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.716978    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-model-cache\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.717093 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.717055    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tmp-dir\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.718798 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.718774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-dshm\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.719016 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.718998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tls-certs\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.725228 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.725208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzcp\" (UniqueName: \"kubernetes.io/projected/4b06a44e-26f8-49e4-99fb-cbd93efbba13-kube-api-access-4lzcp\") pod \"ipp-route-c6c731-kserve-589b65775f-g2r2g\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") " pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.779907 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.779882    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:17.911748 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.911722    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"]
Jul 28 23:00:17.914606 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:00:17.914575    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b06a44e_26f8_49e4_99fb_cbd93efbba13.slice/crio-076baf4d3e0d97c5cee50f8aecf8e70aad0578f551899c90308a416fd8454817 WatchSource:0}: Error finding container 076baf4d3e0d97c5cee50f8aecf8e70aad0578f551899c90308a416fd8454817: Status 404 returned error can't find the container with id 076baf4d3e0d97c5cee50f8aecf8e70aad0578f551899c90308a416fd8454817
Jul 28 23:00:17.982119 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.982068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g" event={"ID":"4b06a44e-26f8-49e4-99fb-cbd93efbba13","Type":"ContainerStarted","Data":"7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8"}
Jul 28 23:00:17.982249 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.982125    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g" event={"ID":"4b06a44e-26f8-49e4-99fb-cbd93efbba13","Type":"ContainerStarted","Data":"076baf4d3e0d97c5cee50f8aecf8e70aad0578f551899c90308a416fd8454817"}
Jul 28 23:00:17.998646 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:17.998594    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g" podStartSLOduration=0.998573834 podStartE2EDuration="998.573834ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:17.996423308 +0000 UTC m=+3020.118771200" watchObservedRunningTime="2026-07-28 23:00:17.998573834 +0000 UTC m=+3020.120921726"
Jul 28 23:00:27.780582 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:27.780539    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:27.781172 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:27.780605    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:27.793415 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:27.793390    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:28.049391 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:28.049308    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:00:49.040721 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:49.040676    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-h2f95"]
Jul 28 23:00:49.044593 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:49.044566    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:00:49.047160 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:49.047135    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-rl6xl\""
Jul 28 23:00:49.053188 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:49.053159    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-h2f95"]
Jul 28 23:00:49.130691 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:49.130646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjs2w\" (UniqueName: \"kubernetes.io/projected/02328577-0272-4e51-8897-8a4d25f70075-kube-api-access-wjs2w\") pod \"kuadrant-operator-catalog-h2f95\" (UID: \"02328577-0272-4e51-8897-8a4d25f70075\") " pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:00:49.232128 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:49.232069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjs2w\" (UniqueName: \"kubernetes.io/projected/02328577-0272-4e51-8897-8a4d25f70075-kube-api-access-wjs2w\") pod \"kuadrant-operator-catalog-h2f95\" (UID: \"02328577-0272-4e51-8897-8a4d25f70075\") " pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:00:49.240087 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:49.240050    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjs2w\" (UniqueName: \"kubernetes.io/projected/02328577-0272-4e51-8897-8a4d25f70075-kube-api-access-wjs2w\") pod \"kuadrant-operator-catalog-h2f95\" (UID: \"02328577-0272-4e51-8897-8a4d25f70075\") " pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:00:49.358341 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:49.358239    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:00:49.500800 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:49.500776    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-h2f95"]
Jul 28 23:00:49.503039 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:00:49.503003    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02328577_0272_4e51_8897_8a4d25f70075.slice/crio-353f3bc7e0492bc91647504051215a0b6ddeda5c1415eb3926c22b6f29ee37ff WatchSource:0}: Error finding container 353f3bc7e0492bc91647504051215a0b6ddeda5c1415eb3926c22b6f29ee37ff: Status 404 returned error can't find the container with id 353f3bc7e0492bc91647504051215a0b6ddeda5c1415eb3926c22b6f29ee37ff
Jul 28 23:00:50.165897 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:50.165821    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-h2f95" event={"ID":"02328577-0272-4e51-8897-8a4d25f70075","Type":"ContainerStarted","Data":"353f3bc7e0492bc91647504051215a0b6ddeda5c1415eb3926c22b6f29ee37ff"}
Jul 28 23:00:52.181017 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:52.180924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-h2f95" event={"ID":"02328577-0272-4e51-8897-8a4d25f70075","Type":"ContainerStarted","Data":"738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007"}
Jul 28 23:00:52.196330 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:52.196274    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-h2f95" podStartSLOduration=0.872265107 podStartE2EDuration="3.196259368s" totalImagesPullingTime="2.323994261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:52.194476436 +0000 UTC m=+3054.316824329" watchObservedRunningTime="2026-07-28 23:00:52.196259368 +0000 UTC m=+3054.318607260"
Jul 28 23:00:59.358723 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:59.358692    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:00:59.361309 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:59.358779    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:00:59.380350 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:00:59.380323    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:01:00.248540 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:00.248509    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:01:00.282383 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:00.282347    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-h2f95"]
Jul 28 23:01:02.237589 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:02.237547    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-h2f95" podUID="02328577-0272-4e51-8897-8a4d25f70075" containerName="registry-server" containerID="cri-o://738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007" gracePeriod=2
Jul 28 23:01:02.248509 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:02.248484    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v_7f7056e9-ea6f-4eff-861d-87d4a41f5e7f/payload-processing/0.log"
Jul 28 23:01:02.370663 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:02.370631    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7g7wm_3e34ca30-b28c-4cca-8472-d23bcf1ddf51/payload-processing/0.log"
Jul 28 23:01:02.478665 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:02.478642    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp_20a41c81-022c-45f3-b193-5ee09aecbf08/payload-processing/0.log"
Jul 28 23:01:02.498043 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:02.497983    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:01:02.570369 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:02.570335    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjs2w\" (UniqueName: \"kubernetes.io/projected/02328577-0272-4e51-8897-8a4d25f70075-kube-api-access-wjs2w\") pod \"02328577-0272-4e51-8897-8a4d25f70075\" (UID: \"02328577-0272-4e51-8897-8a4d25f70075\") "
Jul 28 23:01:02.572320 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:02.572290    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02328577-0272-4e51-8897-8a4d25f70075-kube-api-access-wjs2w" (OuterVolumeSpecName: "kube-api-access-wjs2w") pod "02328577-0272-4e51-8897-8a4d25f70075" (UID: "02328577-0272-4e51-8897-8a4d25f70075"). InnerVolumeSpecName "kube-api-access-wjs2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:01:02.671799 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:02.671763    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjs2w\" (UniqueName: \"kubernetes.io/projected/02328577-0272-4e51-8897-8a4d25f70075-kube-api-access-wjs2w\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:01:02.767357 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:02.767279    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:03.093445 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.093357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:03.244967 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.244924    2560 generic.go:356] "Generic (PLEG): container finished" podID="02328577-0272-4e51-8897-8a4d25f70075" containerID="738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007" exitCode=0
Jul 28 23:01:03.245460 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.244992    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-h2f95"
Jul 28 23:01:03.245460 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.245010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-h2f95" event={"ID":"02328577-0272-4e51-8897-8a4d25f70075","Type":"ContainerDied","Data":"738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007"}
Jul 28 23:01:03.245460 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.245171    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-h2f95" event={"ID":"02328577-0272-4e51-8897-8a4d25f70075","Type":"ContainerDied","Data":"353f3bc7e0492bc91647504051215a0b6ddeda5c1415eb3926c22b6f29ee37ff"}
Jul 28 23:01:03.245460 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.245195    2560 scope.go:117] "RemoveContainer" containerID="738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007"
Jul 28 23:01:03.255999 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.255976    2560 scope.go:117] "RemoveContainer" containerID="738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007"
Jul 28 23:01:03.256319 ip-10-0-129-50 kubenswrapper[2560]: E0728 23:01:03.256292    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007\": container with ID starting with 738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007 not found: ID does not exist" containerID="738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007"
Jul 28 23:01:03.256419 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.256324    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007"} err="failed to get container status \"738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007\": rpc error: code = NotFound desc = could not find container \"738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007\": container with ID starting with 738c49a02bc57b9670ff8e867fe4df6efcf1ef14be03190e2a5f7a4db0baf007 not found: ID does not exist"
Jul 28 23:01:03.266726 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.266699    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-h2f95"]
Jul 28 23:01:03.268988 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.268965    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-h2f95"]
Jul 28 23:01:03.331978 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:03.331945    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:04.408942 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:04.408911    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02328577-0272-4e51-8897-8a4d25f70075" path="/var/lib/kubelet/pods/02328577-0272-4e51-8897-8a4d25f70075/volumes"
Jul 28 23:01:05.785337 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:05.785305    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:05.899903 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:05.899869    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:06.729492 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:06.729446    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:07.154741 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:07.154629    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:07.255457 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:07.255422    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:07.550741 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:07.550695    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:20.474792 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:20.474727    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:21.169968 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.169933    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"]
Jul 28 23:01:21.170442 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.170428    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02328577-0272-4e51-8897-8a4d25f70075" containerName="registry-server"
Jul 28 23:01:21.170503 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.170444    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="02328577-0272-4e51-8897-8a4d25f70075" containerName="registry-server"
Jul 28 23:01:21.170539 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.170515    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="02328577-0272-4e51-8897-8a4d25f70075" containerName="registry-server"
Jul 28 23:01:21.174481 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.174461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"
Jul 28 23:01:21.177458 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.177438    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 23:01:21.187300 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.187273    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"]
Jul 28 23:01:21.258221 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.258162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89b834a-108a-4e24-999f-e4f11775c2b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq\" (UID: \"e89b834a-108a-4e24-999f-e4f11775c2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"
Jul 28 23:01:21.359543 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.359504    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89b834a-108a-4e24-999f-e4f11775c2b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq\" (UID: \"e89b834a-108a-4e24-999f-e4f11775c2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"
Jul 28 23:01:21.360162 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.360141    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89b834a-108a-4e24-999f-e4f11775c2b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq\" (UID: \"e89b834a-108a-4e24-999f-e4f11775c2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"
Jul 28 23:01:21.485030 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.484996    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"
Jul 28 23:01:21.624141 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:21.624111    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"]
Jul 28 23:01:21.626450 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:01:21.626424    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89b834a_108a_4e24_999f_e4f11775c2b9.slice/crio-6a0d9174408bb657f5ff22a0345e4e01c424405a62273198f05f1841243813c7 WatchSource:0}: Error finding container 6a0d9174408bb657f5ff22a0345e4e01c424405a62273198f05f1841243813c7: Status 404 returned error can't find the container with id 6a0d9174408bb657f5ff22a0345e4e01c424405a62273198f05f1841243813c7
Jul 28 23:01:22.354581 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:22.354544    2560 generic.go:356] "Generic (PLEG): container finished" podID="e89b834a-108a-4e24-999f-e4f11775c2b9" containerID="fe51b6bf1852c025c7d8c6dda11492191ee88174ae3f399410873eefb6502f6d" exitCode=0
Jul 28 23:01:22.354736 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:22.354630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq" event={"ID":"e89b834a-108a-4e24-999f-e4f11775c2b9","Type":"ContainerDied","Data":"fe51b6bf1852c025c7d8c6dda11492191ee88174ae3f399410873eefb6502f6d"}
Jul 28 23:01:22.354736 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:22.354687    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq" event={"ID":"e89b834a-108a-4e24-999f-e4f11775c2b9","Type":"ContainerStarted","Data":"6a0d9174408bb657f5ff22a0345e4e01c424405a62273198f05f1841243813c7"}
Jul 28 23:01:23.496236 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:23.496209    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"
Jul 28 23:01:23.579479 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:23.579448    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89b834a-108a-4e24-999f-e4f11775c2b9-maas-api-service-ca\") pod \"e89b834a-108a-4e24-999f-e4f11775c2b9\" (UID: \"e89b834a-108a-4e24-999f-e4f11775c2b9\") "
Jul 28 23:01:23.579833 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:23.579811    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89b834a-108a-4e24-999f-e4f11775c2b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e89b834a-108a-4e24-999f-e4f11775c2b9" (UID: "e89b834a-108a-4e24-999f-e4f11775c2b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:01:23.681132 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:23.681015    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89b834a-108a-4e24-999f-e4f11775c2b9-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:01:24.367250 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:24.367209    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"
Jul 28 23:01:24.367443 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:24.367256    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq" event={"ID":"e89b834a-108a-4e24-999f-e4f11775c2b9","Type":"ContainerDied","Data":"6a0d9174408bb657f5ff22a0345e4e01c424405a62273198f05f1841243813c7"}
Jul 28 23:01:24.367443 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:24.367309    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a0d9174408bb657f5ff22a0345e4e01c424405a62273198f05f1841243813c7"
Jul 28 23:01:28.565415 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:28.565374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:28.664518 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:28.664474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:28.725811 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:28.725771    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"]
Jul 28 23:01:28.727535 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:28.727502    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:28.728332 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:28.728311    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-030bcd-qj8pq"]
Jul 28 23:01:29.088074 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:29.088041    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:30.407007 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:30.406967    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89b834a-108a-4e24-999f-e4f11775c2b9" path="/var/lib/kubelet/pods/e89b834a-108a-4e24-999f-e4f11775c2b9/volumes"
Jul 28 23:01:35.070069 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:35.070036    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:35.174157 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:35.174124    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:35.324569 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:35.324490    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:35.565094 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:35.565062    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:38.950061 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:38.950031    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"]
Jul 28 23:01:38.950497 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:38.950484    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89b834a-108a-4e24-999f-e4f11775c2b9" containerName="revoke-keys"
Jul 28 23:01:38.950541 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:38.950498    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89b834a-108a-4e24-999f-e4f11775c2b9" containerName="revoke-keys"
Jul 28 23:01:38.950597 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:38.950588    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89b834a-108a-4e24-999f-e4f11775c2b9" containerName="revoke-keys"
Jul 28 23:01:38.952897 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:38.952881    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"
Jul 28 23:01:38.955419 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:38.955395    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 23:01:38.969410 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:38.969385    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"]
Jul 28 23:01:39.139858 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:39.139798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c7f93-5ea4-482b-92f1-d3216196158e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4\" (UID: \"f69c7f93-5ea4-482b-92f1-d3216196158e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"
Jul 28 23:01:39.241282 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:39.241233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c7f93-5ea4-482b-92f1-d3216196158e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4\" (UID: \"f69c7f93-5ea4-482b-92f1-d3216196158e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"
Jul 28 23:01:39.241880 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:39.241859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c7f93-5ea4-482b-92f1-d3216196158e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4\" (UID: \"f69c7f93-5ea4-482b-92f1-d3216196158e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"
Jul 28 23:01:39.273728 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:39.273690    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"
Jul 28 23:01:39.415832 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:39.415804    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"]
Jul 28 23:01:39.417513 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:01:39.417487    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69c7f93_5ea4_482b_92f1_d3216196158e.slice/crio-4c86f40fbad8462d96e0ca772a5b1f2b9a561749ab3286d64032ed0ab0e338f5 WatchSource:0}: Error finding container 4c86f40fbad8462d96e0ca772a5b1f2b9a561749ab3286d64032ed0ab0e338f5: Status 404 returned error can't find the container with id 4c86f40fbad8462d96e0ca772a5b1f2b9a561749ab3286d64032ed0ab0e338f5
Jul 28 23:01:39.452356 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:39.452323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4" event={"ID":"f69c7f93-5ea4-482b-92f1-d3216196158e","Type":"ContainerStarted","Data":"4c86f40fbad8462d96e0ca772a5b1f2b9a561749ab3286d64032ed0ab0e338f5"}
Jul 28 23:01:39.798673 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:39.798580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:39.899446 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:39.899417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:40.459354 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:40.459260    2560 generic.go:356] "Generic (PLEG): container finished" podID="f69c7f93-5ea4-482b-92f1-d3216196158e" containerID="a4f1e7f55aa479f0c1865294ca8afc5504010bfae0295fbd4c9b800e7443d52b" exitCode=0
Jul 28 23:01:40.459707 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:40.459349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4" event={"ID":"f69c7f93-5ea4-482b-92f1-d3216196158e","Type":"ContainerDied","Data":"a4f1e7f55aa479f0c1865294ca8afc5504010bfae0295fbd4c9b800e7443d52b"}
Jul 28 23:01:41.597971 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:41.597946    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"
Jul 28 23:01:41.664065 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:41.664032    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c7f93-5ea4-482b-92f1-d3216196158e-maas-api-service-ca\") pod \"f69c7f93-5ea4-482b-92f1-d3216196158e\" (UID: \"f69c7f93-5ea4-482b-92f1-d3216196158e\") "
Jul 28 23:01:41.664437 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:41.664410    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69c7f93-5ea4-482b-92f1-d3216196158e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f69c7f93-5ea4-482b-92f1-d3216196158e" (UID: "f69c7f93-5ea4-482b-92f1-d3216196158e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:01:41.764893 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:41.764813    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c7f93-5ea4-482b-92f1-d3216196158e-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:01:42.471968 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:42.471936    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"
Jul 28 23:01:42.472158 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:42.471986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4" event={"ID":"f69c7f93-5ea4-482b-92f1-d3216196158e","Type":"ContainerDied","Data":"4c86f40fbad8462d96e0ca772a5b1f2b9a561749ab3286d64032ed0ab0e338f5"}
Jul 28 23:01:42.472158 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:42.472045    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c86f40fbad8462d96e0ca772a5b1f2b9a561749ab3286d64032ed0ab0e338f5"
Jul 28 23:01:53.067475 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.067438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:53.068573 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.068547    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql"]
Jul 28 23:01:53.070813 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.070790    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6c731-29754660-d5cql"]
Jul 28 23:01:53.101338 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.101310    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"]
Jul 28 23:01:53.101580 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.101554    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v" podUID="7f7056e9-ea6f-4eff-861d-87d4a41f5e7f" containerName="payload-processing" containerID="cri-o://01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026" gracePeriod=30
Jul 28 23:01:53.165165 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.165128    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:53.236980 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.236945    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"]
Jul 28 23:01:53.241306 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.241270    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:53.241459 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.241285    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c6c731-f9gs4"]
Jul 28 23:01:53.294540 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.294500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:53.364258 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.364233    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 23:01:53.468079 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.468045    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rprc\" (UniqueName: \"kubernetes.io/projected/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-kube-api-access-8rprc\") pod \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\" (UID: \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\") "
Jul 28 23:01:53.468290 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.468093    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-plugins-config-volume\") pod \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\" (UID: \"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f\") "
Jul 28 23:01:53.468673 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.468475    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f7056e9-ea6f-4eff-861d-87d4a41f5e7f" (UID: "7f7056e9-ea6f-4eff-861d-87d4a41f5e7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:01:53.468673 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.468641    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:01:53.470266 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.470243    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-kube-api-access-8rprc" (OuterVolumeSpecName: "kube-api-access-8rprc") pod "7f7056e9-ea6f-4eff-861d-87d4a41f5e7f" (UID: "7f7056e9-ea6f-4eff-861d-87d4a41f5e7f"). InnerVolumeSpecName "kube-api-access-8rprc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:01:53.548189 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.548155    2560 generic.go:356] "Generic (PLEG): container finished" podID="7f7056e9-ea6f-4eff-861d-87d4a41f5e7f" containerID="01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026" exitCode=0
Jul 28 23:01:53.548342 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.548196    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"
Jul 28 23:01:53.548342 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.548240    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v" event={"ID":"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f","Type":"ContainerDied","Data":"01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026"}
Jul 28 23:01:53.548342 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.548299    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v" event={"ID":"7f7056e9-ea6f-4eff-861d-87d4a41f5e7f","Type":"ContainerDied","Data":"ea47789e562355ee83cd936de310de1ecdbfc52414743bda76af8b165e85d8cf"}
Jul 28 23:01:53.548342 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.548328    2560 scope.go:117] "RemoveContainer" containerID="01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026"
Jul 28 23:01:53.557676 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.557653    2560 scope.go:117] "RemoveContainer" containerID="01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026"
Jul 28 23:01:53.557943 ip-10-0-129-50 kubenswrapper[2560]: E0728 23:01:53.557923    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026\": container with ID starting with 01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026 not found: ID does not exist" containerID="01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026"
Jul 28 23:01:53.557992 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.557952    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026"} err="failed to get container status \"01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026\": rpc error: code = NotFound desc = could not find container \"01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026\": container with ID starting with 01518f30961751efe3f87d5d5d0c02e1fbb90c5ece1908a236cc6d195f54e026 not found: ID does not exist"
Jul 28 23:01:53.569675 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.569648    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rprc\" (UniqueName: \"kubernetes.io/projected/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f-kube-api-access-8rprc\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:01:53.570765 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.570733    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"]
Jul 28 23:01:53.577317 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.577294    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6c731-c7f4879bd-9sk2v"]
Jul 28 23:01:53.605155 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:53.605047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:01:54.410841 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:54.410802    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c78c7da-bb55-4c16-a609-130a158d0228" path="/var/lib/kubelet/pods/4c78c7da-bb55-4c16-a609-130a158d0228/volumes"
Jul 28 23:01:54.411347 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:54.411329    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7056e9-ea6f-4eff-861d-87d4a41f5e7f" path="/var/lib/kubelet/pods/7f7056e9-ea6f-4eff-861d-87d4a41f5e7f/volumes"
Jul 28 23:01:54.411678 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:54.411658    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f69c7f93-5ea4-482b-92f1-d3216196158e" path="/var/lib/kubelet/pods/f69c7f93-5ea4-482b-92f1-d3216196158e/volumes"
Jul 28 23:01:58.587528 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:58.587496    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"]
Jul 28 23:01:58.629699 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:01:58.629666    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:13.607119 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.607032    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g" podUID="4b06a44e-26f8-49e4-99fb-cbd93efbba13" containerName="main" containerID="cri-o://7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8" gracePeriod=45
Jul 28 23:02:13.860149 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.860080    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:02:13.966249 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966210    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lzcp\" (UniqueName: \"kubernetes.io/projected/4b06a44e-26f8-49e4-99fb-cbd93efbba13-kube-api-access-4lzcp\") pod \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") "
Jul 28 23:02:13.966444 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966290    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-home\") pod \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") "
Jul 28 23:02:13.966444 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966344    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-dshm\") pod \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") "
Jul 28 23:02:13.966444 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966363    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tls-certs\") pod \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") "
Jul 28 23:02:13.966444 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966406    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tmp-dir\") pod \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") "
Jul 28 23:02:13.966669 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-model-cache\") pod \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\" (UID: \"4b06a44e-26f8-49e4-99fb-cbd93efbba13\") "
Jul 28 23:02:13.966669 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966510    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-home" (OuterVolumeSpecName: "home") pod "4b06a44e-26f8-49e4-99fb-cbd93efbba13" (UID: "4b06a44e-26f8-49e4-99fb-cbd93efbba13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:02:13.966779 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966710    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b06a44e-26f8-49e4-99fb-cbd93efbba13" (UID: "4b06a44e-26f8-49e4-99fb-cbd93efbba13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:02:13.966839 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966788    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b06a44e-26f8-49e4-99fb-cbd93efbba13" (UID: "4b06a44e-26f8-49e4-99fb-cbd93efbba13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:02:13.966884 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966842    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tmp-dir\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:13.966884 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.966861    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-home\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:13.968704 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.968672    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-dshm" (OuterVolumeSpecName: "dshm") pod "4b06a44e-26f8-49e4-99fb-cbd93efbba13" (UID: "4b06a44e-26f8-49e4-99fb-cbd93efbba13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:02:13.968704 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.968681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b06a44e-26f8-49e4-99fb-cbd93efbba13" (UID: "4b06a44e-26f8-49e4-99fb-cbd93efbba13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:02:13.968704 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:13.968682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b06a44e-26f8-49e4-99fb-cbd93efbba13-kube-api-access-4lzcp" (OuterVolumeSpecName: "kube-api-access-4lzcp") pod "4b06a44e-26f8-49e4-99fb-cbd93efbba13" (UID: "4b06a44e-26f8-49e4-99fb-cbd93efbba13"). InnerVolumeSpecName "kube-api-access-4lzcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:14.068088 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.068054    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:14.068088 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.068084    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lzcp\" (UniqueName: \"kubernetes.io/projected/4b06a44e-26f8-49e4-99fb-cbd93efbba13-kube-api-access-4lzcp\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:14.068314 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.068093    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b06a44e-26f8-49e4-99fb-cbd93efbba13-dshm\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:14.068314 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.068125    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b06a44e-26f8-49e4-99fb-cbd93efbba13-tls-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:14.667721 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.667636    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b06a44e-26f8-49e4-99fb-cbd93efbba13" containerID="7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8" exitCode=0
Jul 28 23:02:14.667721 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.667683    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"
Jul 28 23:02:14.668188 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.667721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g" event={"ID":"4b06a44e-26f8-49e4-99fb-cbd93efbba13","Type":"ContainerDied","Data":"7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8"}
Jul 28 23:02:14.668188 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.667765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g" event={"ID":"4b06a44e-26f8-49e4-99fb-cbd93efbba13","Type":"ContainerDied","Data":"076baf4d3e0d97c5cee50f8aecf8e70aad0578f551899c90308a416fd8454817"}
Jul 28 23:02:14.668188 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.667786    2560 scope.go:117] "RemoveContainer" containerID="7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8"
Jul 28 23:02:14.676854 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.676834    2560 scope.go:117] "RemoveContainer" containerID="7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8"
Jul 28 23:02:14.677122 ip-10-0-129-50 kubenswrapper[2560]: E0728 23:02:14.677085    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8\": container with ID starting with 7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8 not found: ID does not exist" containerID="7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8"
Jul 28 23:02:14.677206 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.677129    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8"} err="failed to get container status \"7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8\": rpc error: code = NotFound desc = could not find container \"7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8\": container with ID starting with 7da177b1ac2cad00a88811e3b333a28db6148a09c9ae2b30905f8855ae4616f8 not found: ID does not exist"
Jul 28 23:02:14.683379 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.683356    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"]
Jul 28 23:02:14.687093 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:14.687072    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c6c731/ipp-route-c6c731-kserve-589b65775f-g2r2g"]
Jul 28 23:02:16.415142 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:16.415086    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b06a44e-26f8-49e4-99fb-cbd93efbba13" path="/var/lib/kubelet/pods/4b06a44e-26f8-49e4-99fb-cbd93efbba13/volumes"
Jul 28 23:02:33.011182 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:33.011060    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:35.164888 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:35.164854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:35.630804 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:35.630758    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:41.288089 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.288048    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:41.397319 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.391360    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:41.581658 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.581554    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:41.806167 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.806132    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:41.929823 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.929731    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"]
Jul 28 23:02:41.930320 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.930300    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b06a44e-26f8-49e4-99fb-cbd93efbba13" containerName="main"
Jul 28 23:02:41.930320 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.930321    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b06a44e-26f8-49e4-99fb-cbd93efbba13" containerName="main"
Jul 28 23:02:41.930494 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.930333    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7056e9-ea6f-4eff-861d-87d4a41f5e7f" containerName="payload-processing"
Jul 28 23:02:41.930494 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.930342    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7056e9-ea6f-4eff-861d-87d4a41f5e7f" containerName="payload-processing"
Jul 28 23:02:41.930494 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.930362    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f69c7f93-5ea4-482b-92f1-d3216196158e" containerName="revoke-keys"
Jul 28 23:02:41.930494 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.930370    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f69c7f93-5ea4-482b-92f1-d3216196158e" containerName="revoke-keys"
Jul 28 23:02:41.930494 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.930480    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7056e9-ea6f-4eff-861d-87d4a41f5e7f" containerName="payload-processing"
Jul 28 23:02:41.930729 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.930496    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f69c7f93-5ea4-482b-92f1-d3216196158e" containerName="revoke-keys"
Jul 28 23:02:41.930729 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.930507    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b06a44e-26f8-49e4-99fb-cbd93efbba13" containerName="main"
Jul 28 23:02:41.932651 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.932629    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"
Jul 28 23:02:41.935043 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.935019    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 23:02:41.941171 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:41.940738    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"]
Jul 28 23:02:42.044397 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:42.044357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbcbb7b-8912-49a0-9889-b6470e3c9788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb\" (UID: \"0dbcbb7b-8912-49a0-9889-b6470e3c9788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"
Jul 28 23:02:42.145571 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:42.145528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbcbb7b-8912-49a0-9889-b6470e3c9788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb\" (UID: \"0dbcbb7b-8912-49a0-9889-b6470e3c9788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"
Jul 28 23:02:42.146249 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:42.146228    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbcbb7b-8912-49a0-9889-b6470e3c9788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb\" (UID: \"0dbcbb7b-8912-49a0-9889-b6470e3c9788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"
Jul 28 23:02:42.245712 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:42.245681    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"
Jul 28 23:02:42.385419 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:42.385395    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"]
Jul 28 23:02:42.387817 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:02:42.387787    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbcbb7b_8912_49a0_9889_b6470e3c9788.slice/crio-4b2356421a6bf172d3f7fce68c4f51470f393a19373e159822b25c520a91f028 WatchSource:0}: Error finding container 4b2356421a6bf172d3f7fce68c4f51470f393a19373e159822b25c520a91f028: Status 404 returned error can't find the container with id 4b2356421a6bf172d3f7fce68c4f51470f393a19373e159822b25c520a91f028
Jul 28 23:02:42.830123 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:42.830062    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb" event={"ID":"0dbcbb7b-8912-49a0-9889-b6470e3c9788","Type":"ContainerStarted","Data":"51a4aaa3706540c45516ea164994b55b0454779f37ff5520608896f34b9f2c3a"}
Jul 28 23:02:42.830331 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:42.830148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb" event={"ID":"0dbcbb7b-8912-49a0-9889-b6470e3c9788","Type":"ContainerStarted","Data":"4b2356421a6bf172d3f7fce68c4f51470f393a19373e159822b25c520a91f028"}
Jul 28 23:02:42.844366 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:42.844321    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb" podStartSLOduration=1.844305829 podStartE2EDuration="1.844305829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:02:42.843184333 +0000 UTC m=+3164.965532236" watchObservedRunningTime="2026-07-28 23:02:42.844305829 +0000 UTC m=+3164.966653724"
Jul 28 23:02:43.837769 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:43.837738    2560 generic.go:356] "Generic (PLEG): container finished" podID="0dbcbb7b-8912-49a0-9889-b6470e3c9788" containerID="51a4aaa3706540c45516ea164994b55b0454779f37ff5520608896f34b9f2c3a" exitCode=0
Jul 28 23:02:43.837769 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:43.837773    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb" event={"ID":"0dbcbb7b-8912-49a0-9889-b6470e3c9788","Type":"ContainerDied","Data":"51a4aaa3706540c45516ea164994b55b0454779f37ff5520608896f34b9f2c3a"}
Jul 28 23:02:44.981022 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:44.980995    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"
Jul 28 23:02:45.072553 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:45.072514    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbcbb7b-8912-49a0-9889-b6470e3c9788-maas-api-service-ca\") pod \"0dbcbb7b-8912-49a0-9889-b6470e3c9788\" (UID: \"0dbcbb7b-8912-49a0-9889-b6470e3c9788\") "
Jul 28 23:02:45.072887 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:45.072862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbcbb7b-8912-49a0-9889-b6470e3c9788-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dbcbb7b-8912-49a0-9889-b6470e3c9788" (UID: "0dbcbb7b-8912-49a0-9889-b6470e3c9788"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:45.173868 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:45.173785    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbcbb7b-8912-49a0-9889-b6470e3c9788-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:45.850027 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:45.850002    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"
Jul 28 23:02:45.850027 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:45.850021    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb" event={"ID":"0dbcbb7b-8912-49a0-9889-b6470e3c9788","Type":"ContainerDied","Data":"4b2356421a6bf172d3f7fce68c4f51470f393a19373e159822b25c520a91f028"}
Jul 28 23:02:45.850274 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:45.850066    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b2356421a6bf172d3f7fce68c4f51470f393a19373e159822b25c520a91f028"
Jul 28 23:02:47.590391 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.590355    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"]
Jul 28 23:02:47.590853 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.590607    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd" podUID="87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe" containerName="maas-api" containerID="cri-o://7b77cf7dbc21327cdeecf5e7ba2413181ed85d70d1737934f71ef94b2b5d59c9" gracePeriod=30
Jul 28 23:02:47.616341 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.616306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:47.619035 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.619002    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5"]
Jul 28 23:02:47.621452 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.621427    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6fb213-29754660-f22r5"]
Jul 28 23:02:47.644716 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.644677    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"]
Jul 28 23:02:47.644951 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.644925    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp" podUID="20a41c81-022c-45f3-b193-5ee09aecbf08" containerName="payload-processing" containerID="cri-o://5768f977ea49b1b240b51e852efc0360aeb12155abd6cec7edc986533a6f3dda" gracePeriod=30
Jul 28 23:02:47.652376 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.652350    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"]
Jul 28 23:02:47.652606 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.652558    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v" podUID="44f02181-8930-4262-8a8b-d5df9c2936f1" containerName="payload-pre-processing" containerID="cri-o://84b468a332acdf8b19d5ac2a63a1ec902fd118e2f2a00d276ceabcc726b6fc5e" gracePeriod=30
Jul 28 23:02:47.690482 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.690452    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:47.748674 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.748642    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:47.782191 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.782142    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"]
Jul 28 23:02:47.786085 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.786050    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6fb213-8b4fb"]
Jul 28 23:02:47.807954 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.806600    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:47.866017 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.865989    2560 generic.go:356] "Generic (PLEG): container finished" podID="87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe" containerID="7b77cf7dbc21327cdeecf5e7ba2413181ed85d70d1737934f71ef94b2b5d59c9" exitCode=0
Jul 28 23:02:47.866196 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.866063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd" event={"ID":"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe","Type":"ContainerDied","Data":"7b77cf7dbc21327cdeecf5e7ba2413181ed85d70d1737934f71ef94b2b5d59c9"}
Jul 28 23:02:47.867804 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.867783    2560 generic.go:356] "Generic (PLEG): container finished" podID="20a41c81-022c-45f3-b193-5ee09aecbf08" containerID="5768f977ea49b1b240b51e852efc0360aeb12155abd6cec7edc986533a6f3dda" exitCode=0
Jul 28 23:02:47.867935 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.867860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp" event={"ID":"20a41c81-022c-45f3-b193-5ee09aecbf08","Type":"ContainerDied","Data":"5768f977ea49b1b240b51e852efc0360aeb12155abd6cec7edc986533a6f3dda"}
Jul 28 23:02:47.869711 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.869690    2560 generic.go:356] "Generic (PLEG): container finished" podID="44f02181-8930-4262-8a8b-d5df9c2936f1" containerID="84b468a332acdf8b19d5ac2a63a1ec902fd118e2f2a00d276ceabcc726b6fc5e" exitCode=0
Jul 28 23:02:47.869828 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.869767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v" event={"ID":"44f02181-8930-4262-8a8b-d5df9c2936f1","Type":"ContainerDied","Data":"84b468a332acdf8b19d5ac2a63a1ec902fd118e2f2a00d276ceabcc726b6fc5e"}
Jul 28 23:02:47.915346 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.915306    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 23:02:47.933228 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.933200    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 23:02:47.950691 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:47.950665    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 23:02:48.001326 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001297    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a41c81-022c-45f3-b193-5ee09aecbf08-plugins-config-volume\") pod \"20a41c81-022c-45f3-b193-5ee09aecbf08\" (UID: \"20a41c81-022c-45f3-b193-5ee09aecbf08\") "
Jul 28 23:02:48.001494 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001337    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f02181-8930-4262-8a8b-d5df9c2936f1-plugins-config-volume\") pod \"44f02181-8930-4262-8a8b-d5df9c2936f1\" (UID: \"44f02181-8930-4262-8a8b-d5df9c2936f1\") "
Jul 28 23:02:48.001494 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001462    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gqpg\" (UniqueName: \"kubernetes.io/projected/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-kube-api-access-6gqpg\") pod \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") "
Jul 28 23:02:48.001609 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001507    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm5vw\" (UniqueName: \"kubernetes.io/projected/20a41c81-022c-45f3-b193-5ee09aecbf08-kube-api-access-gm5vw\") pod \"20a41c81-022c-45f3-b193-5ee09aecbf08\" (UID: \"20a41c81-022c-45f3-b193-5ee09aecbf08\") "
Jul 28 23:02:48.001609 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001555    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg9sq\" (UniqueName: \"kubernetes.io/projected/44f02181-8930-4262-8a8b-d5df9c2936f1-kube-api-access-jg9sq\") pod \"44f02181-8930-4262-8a8b-d5df9c2936f1\" (UID: \"44f02181-8930-4262-8a8b-d5df9c2936f1\") "
Jul 28 23:02:48.001609 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001598    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-openshift-service-ca\") pod \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") "
Jul 28 23:02:48.001772 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001651    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-maas-api-tls\") pod \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\" (UID: \"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe\") "
Jul 28 23:02:48.001772 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001655    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f02181-8930-4262-8a8b-d5df9c2936f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44f02181-8930-4262-8a8b-d5df9c2936f1" (UID: "44f02181-8930-4262-8a8b-d5df9c2936f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:48.001772 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001726    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a41c81-022c-45f3-b193-5ee09aecbf08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20a41c81-022c-45f3-b193-5ee09aecbf08" (UID: "20a41c81-022c-45f3-b193-5ee09aecbf08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:48.001997 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001980    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a41c81-022c-45f3-b193-5ee09aecbf08-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:48.002057 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.001998    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f02181-8930-4262-8a8b-d5df9c2936f1-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:48.002125 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.002091    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe" (UID: "87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:48.003773 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.003744    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-kube-api-access-6gqpg" (OuterVolumeSpecName: "kube-api-access-6gqpg") pod "87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe" (UID: "87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe"). InnerVolumeSpecName "kube-api-access-6gqpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:48.003773 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.003768    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a41c81-022c-45f3-b193-5ee09aecbf08-kube-api-access-gm5vw" (OuterVolumeSpecName: "kube-api-access-gm5vw") pod "20a41c81-022c-45f3-b193-5ee09aecbf08" (UID: "20a41c81-022c-45f3-b193-5ee09aecbf08"). InnerVolumeSpecName "kube-api-access-gm5vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:48.003931 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.003769    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe" (UID: "87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:02:48.003931 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.003785    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f02181-8930-4262-8a8b-d5df9c2936f1-kube-api-access-jg9sq" (OuterVolumeSpecName: "kube-api-access-jg9sq") pod "44f02181-8930-4262-8a8b-d5df9c2936f1" (UID: "44f02181-8930-4262-8a8b-d5df9c2936f1"). InnerVolumeSpecName "kube-api-access-jg9sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:48.103030 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.102920    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqpg\" (UniqueName: \"kubernetes.io/projected/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-kube-api-access-6gqpg\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:48.103030 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.102961    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm5vw\" (UniqueName: \"kubernetes.io/projected/20a41c81-022c-45f3-b193-5ee09aecbf08-kube-api-access-gm5vw\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:48.103030 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.102970    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg9sq\" (UniqueName: \"kubernetes.io/projected/44f02181-8930-4262-8a8b-d5df9c2936f1-kube-api-access-jg9sq\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:48.103030 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.102979    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-openshift-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:48.103030 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.102988    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe-maas-api-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:48.159051 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.159019    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:48.407686 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.407604    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dbcbb7b-8912-49a0-9889-b6470e3c9788" path="/var/lib/kubelet/pods/0dbcbb7b-8912-49a0-9889-b6470e3c9788/volumes"
Jul 28 23:02:48.407907 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.407894    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee" path="/var/lib/kubelet/pods/19d8eab9-7eed-4cf3-8ca4-1ea57b2b36ee/volumes"
Jul 28 23:02:48.877242 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.877204    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"
Jul 28 23:02:48.877242 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.877234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd" event={"ID":"87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe","Type":"ContainerDied","Data":"7c4941aac1c7862db156ce19b08c220e89ad0c51bc97a35ab9b58e60350a9240"}
Jul 28 23:02:48.877777 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.877308    2560 scope.go:117] "RemoveContainer" containerID="7b77cf7dbc21327cdeecf5e7ba2413181ed85d70d1737934f71ef94b2b5d59c9"
Jul 28 23:02:48.878729 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.878593    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"
Jul 28 23:02:48.878729 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.878612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp" event={"ID":"20a41c81-022c-45f3-b193-5ee09aecbf08","Type":"ContainerDied","Data":"b89bb73f38689b2f004667dde69ca6b47ba4b8d2a9d4f4c5bd914c8bb685cc72"}
Jul 28 23:02:48.880191 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.880172    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"
Jul 28 23:02:48.880315 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.880212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v" event={"ID":"44f02181-8930-4262-8a8b-d5df9c2936f1","Type":"ContainerDied","Data":"1976d0eea57b98f8590498a7cd40b67da94e3b032e073b043938a19660f2f4d6"}
Jul 28 23:02:48.888217 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.888199    2560 scope.go:117] "RemoveContainer" containerID="5768f977ea49b1b240b51e852efc0360aeb12155abd6cec7edc986533a6f3dda"
Jul 28 23:02:48.896298 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.896282    2560 scope.go:117] "RemoveContainer" containerID="84b468a332acdf8b19d5ac2a63a1ec902fd118e2f2a00d276ceabcc726b6fc5e"
Jul 28 23:02:48.899563 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.899540    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"]
Jul 28 23:02:48.902016 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.901992    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6fb213-6b7767b8b8-hvl8v"]
Jul 28 23:02:48.912606 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.912571    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"]
Jul 28 23:02:48.915749 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.915726    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6fb213-57b57c6fb4-hglbd"]
Jul 28 23:02:48.925258 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.925234    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"]
Jul 28 23:02:48.928572 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:48.928552    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6fb213-585978fff6-hf5dp"]
Jul 28 23:02:50.407303 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:50.407268    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a41c81-022c-45f3-b193-5ee09aecbf08" path="/var/lib/kubelet/pods/20a41c81-022c-45f3-b193-5ee09aecbf08/volumes"
Jul 28 23:02:50.407687 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:50.407636    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f02181-8930-4262-8a8b-d5df9c2936f1" path="/var/lib/kubelet/pods/44f02181-8930-4262-8a8b-d5df9c2936f1/volumes"
Jul 28 23:02:50.407973 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:50.407960    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe" path="/var/lib/kubelet/pods/87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe/volumes"
Jul 28 23:02:54.002618 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.002576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:54.102448 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.102412    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:54.262773 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.262673    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:54.508545 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.508509    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:54.643549 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.643454    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"]
Jul 28 23:02:54.643959 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.643938    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dbcbb7b-8912-49a0-9889-b6470e3c9788" containerName="revoke-keys"
Jul 28 23:02:54.644057 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.643961    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dbcbb7b-8912-49a0-9889-b6470e3c9788" containerName="revoke-keys"
Jul 28 23:02:54.644057 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644007    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a41c81-022c-45f3-b193-5ee09aecbf08" containerName="payload-processing"
Jul 28 23:02:54.644057 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644017    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a41c81-022c-45f3-b193-5ee09aecbf08" containerName="payload-processing"
Jul 28 23:02:54.644057 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644032    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f02181-8930-4262-8a8b-d5df9c2936f1" containerName="payload-pre-processing"
Jul 28 23:02:54.644057 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644041    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f02181-8930-4262-8a8b-d5df9c2936f1" containerName="payload-pre-processing"
Jul 28 23:02:54.644326 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644059    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe" containerName="maas-api"
Jul 28 23:02:54.644326 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644067    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe" containerName="maas-api"
Jul 28 23:02:54.644326 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644186    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87c6c0bf-a84e-4dda-b5da-74e5ff8ee9fe" containerName="maas-api"
Jul 28 23:02:54.644326 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644201    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a41c81-022c-45f3-b193-5ee09aecbf08" containerName="payload-processing"
Jul 28 23:02:54.644326 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644214    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f02181-8930-4262-8a8b-d5df9c2936f1" containerName="payload-pre-processing"
Jul 28 23:02:54.644326 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.644223    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dbcbb7b-8912-49a0-9889-b6470e3c9788" containerName="revoke-keys"
Jul 28 23:02:54.649160 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.649138    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"
Jul 28 23:02:54.652833 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.652810    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 23:02:54.661412 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.661384    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"]
Jul 28 23:02:54.764024 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.763988    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2af918a-9e1f-4254-b574-cc928a56fd0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp\" (UID: \"b2af918a-9e1f-4254-b574-cc928a56fd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"
Jul 28 23:02:54.864717 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.864679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2af918a-9e1f-4254-b574-cc928a56fd0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp\" (UID: \"b2af918a-9e1f-4254-b574-cc928a56fd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"
Jul 28 23:02:54.865312 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.865294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2af918a-9e1f-4254-b574-cc928a56fd0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp\" (UID: \"b2af918a-9e1f-4254-b574-cc928a56fd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"
Jul 28 23:02:54.960660 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:54.960580    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"
Jul 28 23:02:55.099085 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:55.099060    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"]
Jul 28 23:02:55.101705 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:02:55.101679    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2af918a_9e1f_4254_b574_cc928a56fd0a.slice/crio-5ae9b169b134129759b18dea89058717b7f02a3038c0f14336975937e597988d WatchSource:0}: Error finding container 5ae9b169b134129759b18dea89058717b7f02a3038c0f14336975937e597988d: Status 404 returned error can't find the container with id 5ae9b169b134129759b18dea89058717b7f02a3038c0f14336975937e597988d
Jul 28 23:02:55.922980 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:55.922945    2560 generic.go:356] "Generic (PLEG): container finished" podID="b2af918a-9e1f-4254-b574-cc928a56fd0a" containerID="97cdb72418563ee7ad654f21d25e008431ef91846432c6ef33b8badba711fb5b" exitCode=0
Jul 28 23:02:55.923181 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:55.923028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp" event={"ID":"b2af918a-9e1f-4254-b574-cc928a56fd0a","Type":"ContainerDied","Data":"97cdb72418563ee7ad654f21d25e008431ef91846432c6ef33b8badba711fb5b"}
Jul 28 23:02:55.923181 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:55.923080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp" event={"ID":"b2af918a-9e1f-4254-b574-cc928a56fd0a","Type":"ContainerStarted","Data":"5ae9b169b134129759b18dea89058717b7f02a3038c0f14336975937e597988d"}
Jul 28 23:02:57.063988 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:57.063961    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"
Jul 28 23:02:57.188432 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:57.188338    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2af918a-9e1f-4254-b574-cc928a56fd0a-maas-api-service-ca\") pod \"b2af918a-9e1f-4254-b574-cc928a56fd0a\" (UID: \"b2af918a-9e1f-4254-b574-cc928a56fd0a\") "
Jul 28 23:02:57.188698 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:57.188677    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2af918a-9e1f-4254-b574-cc928a56fd0a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2af918a-9e1f-4254-b574-cc928a56fd0a" (UID: "b2af918a-9e1f-4254-b574-cc928a56fd0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:57.289896 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:57.289859    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2af918a-9e1f-4254-b574-cc928a56fd0a-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:02:57.935360 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:57.935332    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"
Jul 28 23:02:57.935562 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:57.935369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp" event={"ID":"b2af918a-9e1f-4254-b574-cc928a56fd0a","Type":"ContainerDied","Data":"5ae9b169b134129759b18dea89058717b7f02a3038c0f14336975937e597988d"}
Jul 28 23:02:57.935562 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:57.935415    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ae9b169b134129759b18dea89058717b7f02a3038c0f14336975937e597988d"
Jul 28 23:02:59.886778 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:59.886725    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:02:59.892217 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:59.892189    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk"]
Jul 28 23:02:59.894996 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:59.894971    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a134df-29754660-7pwfk"]
Jul 28 23:02:59.926769 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:59.926735    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"]
Jul 28 23:02:59.927226 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:59.927196    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9" podUID="5bcc4db3-9814-4b43-a4a8-96f0232c052d" containerName="payload-pre-processing" containerID="cri-o://36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4" gracePeriod=30
Jul 28 23:02:59.986906 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:02:59.986873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:00.045112 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.045064    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"]
Jul 28 23:03:00.048958 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.048925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:00.049823 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.049796    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a134df-gz5sp"]
Jul 28 23:03:00.186394 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.186368    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 23:03:00.320147 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.320093    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcc4db3-9814-4b43-a4a8-96f0232c052d-plugins-config-volume\") pod \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\" (UID: \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\") "
Jul 28 23:03:00.320307 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.320158    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ks9x\" (UniqueName: \"kubernetes.io/projected/5bcc4db3-9814-4b43-a4a8-96f0232c052d-kube-api-access-4ks9x\") pod \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\" (UID: \"5bcc4db3-9814-4b43-a4a8-96f0232c052d\") "
Jul 28 23:03:00.320519 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.320493    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bcc4db3-9814-4b43-a4a8-96f0232c052d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5bcc4db3-9814-4b43-a4a8-96f0232c052d" (UID: "5bcc4db3-9814-4b43-a4a8-96f0232c052d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:03:00.322255 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.322234    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bcc4db3-9814-4b43-a4a8-96f0232c052d-kube-api-access-4ks9x" (OuterVolumeSpecName: "kube-api-access-4ks9x") pod "5bcc4db3-9814-4b43-a4a8-96f0232c052d" (UID: "5bcc4db3-9814-4b43-a4a8-96f0232c052d"). InnerVolumeSpecName "kube-api-access-4ks9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:03:00.408294 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.408258    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11b5963a-f1ea-43c9-8082-442f485e958f" path="/var/lib/kubelet/pods/11b5963a-f1ea-43c9-8082-442f485e958f/volumes"
Jul 28 23:03:00.408688 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.408667    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2af918a-9e1f-4254-b574-cc928a56fd0a" path="/var/lib/kubelet/pods/b2af918a-9e1f-4254-b574-cc928a56fd0a/volumes"
Jul 28 23:03:00.409389 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.409363    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:00.421069 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.421046    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcc4db3-9814-4b43-a4a8-96f0232c052d-plugins-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:03:00.421192 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.421070    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ks9x\" (UniqueName: \"kubernetes.io/projected/5bcc4db3-9814-4b43-a4a8-96f0232c052d-kube-api-access-4ks9x\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:03:00.962864 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.962831    2560 generic.go:356] "Generic (PLEG): container finished" podID="5bcc4db3-9814-4b43-a4a8-96f0232c052d" containerID="36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4" exitCode=0
Jul 28 23:03:00.963372 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.962869    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"
Jul 28 23:03:00.963372 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.962920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9" event={"ID":"5bcc4db3-9814-4b43-a4a8-96f0232c052d","Type":"ContainerDied","Data":"36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4"}
Jul 28 23:03:00.963372 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.962983    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9" event={"ID":"5bcc4db3-9814-4b43-a4a8-96f0232c052d","Type":"ContainerDied","Data":"8b0310916d8ad90056ef7b60b587fef695656f6855e938437afddffa644c5b03"}
Jul 28 23:03:00.963372 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.963019    2560 scope.go:117] "RemoveContainer" containerID="36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4"
Jul 28 23:03:00.972139 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.972119    2560 scope.go:117] "RemoveContainer" containerID="36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4"
Jul 28 23:03:00.972463 ip-10-0-129-50 kubenswrapper[2560]: E0728 23:03:00.972444    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4\": container with ID starting with 36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4 not found: ID does not exist" containerID="36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4"
Jul 28 23:03:00.972509 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.972469    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4"} err="failed to get container status \"36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4\": rpc error: code = NotFound desc = could not find container \"36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4\": container with ID starting with 36712746085a8da03b170bde63cd8879dcb479701b27995fe45697a24c84bcc4 not found: ID does not exist"
Jul 28 23:03:00.981848 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.981818    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"]
Jul 28 23:03:00.987066 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:00.987037    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a134df-855b99bbd4-td9b9"]
Jul 28 23:03:02.407316 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:02.407283    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bcc4db3-9814-4b43-a4a8-96f0232c052d" path="/var/lib/kubelet/pods/5bcc4db3-9814-4b43-a4a8-96f0232c052d/volumes"
Jul 28 23:03:06.273596 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.273562    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:06.332033 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.331994    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:06.529615 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.529514    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:06.748373 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.748333    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:06.879455 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.879374    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"]
Jul 28 23:03:06.879788 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.879775    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2af918a-9e1f-4254-b574-cc928a56fd0a" containerName="revoke-keys"
Jul 28 23:03:06.879835 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.879789    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2af918a-9e1f-4254-b574-cc928a56fd0a" containerName="revoke-keys"
Jul 28 23:03:06.879835 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.879802    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bcc4db3-9814-4b43-a4a8-96f0232c052d" containerName="payload-pre-processing"
Jul 28 23:03:06.879835 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.879808    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bcc4db3-9814-4b43-a4a8-96f0232c052d" containerName="payload-pre-processing"
Jul 28 23:03:06.879935 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.879910    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2af918a-9e1f-4254-b574-cc928a56fd0a" containerName="revoke-keys"
Jul 28 23:03:06.879935 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.879921    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bcc4db3-9814-4b43-a4a8-96f0232c052d" containerName="payload-pre-processing"
Jul 28 23:03:06.884721 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.884703    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"
Jul 28 23:03:06.887232 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.887211    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\""
Jul 28 23:03:06.890821 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.890796    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"]
Jul 28 23:03:06.982979 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:06.982936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3632e237-bca4-47f1-afd8-06b62910ad82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf\" (UID: \"3632e237-bca4-47f1-afd8-06b62910ad82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"
Jul 28 23:03:07.084406 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:07.084367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3632e237-bca4-47f1-afd8-06b62910ad82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf\" (UID: \"3632e237-bca4-47f1-afd8-06b62910ad82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"
Jul 28 23:03:07.084992 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:07.084972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3632e237-bca4-47f1-afd8-06b62910ad82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf\" (UID: \"3632e237-bca4-47f1-afd8-06b62910ad82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"
Jul 28 23:03:07.197691 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:07.197597    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"
Jul 28 23:03:07.340954 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:07.340925    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"]
Jul 28 23:03:07.342643 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:03:07.342618    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3632e237_bca4_47f1_afd8_06b62910ad82.slice/crio-f57f3627739b3cf6ef5e542e40e621165420df29ce719c71adba6138deb5cf36 WatchSource:0}: Error finding container f57f3627739b3cf6ef5e542e40e621165420df29ce719c71adba6138deb5cf36: Status 404 returned error can't find the container with id f57f3627739b3cf6ef5e542e40e621165420df29ce719c71adba6138deb5cf36
Jul 28 23:03:08.006160 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:08.006056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf" event={"ID":"3632e237-bca4-47f1-afd8-06b62910ad82","Type":"ContainerStarted","Data":"40c01bba111039616587bb8c60339ff99f1be55b901746bd96b60e6c486647a3"}
Jul 28 23:03:08.006160 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:08.006142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf" event={"ID":"3632e237-bca4-47f1-afd8-06b62910ad82","Type":"ContainerStarted","Data":"f57f3627739b3cf6ef5e542e40e621165420df29ce719c71adba6138deb5cf36"}
Jul 28 23:03:09.013354 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:09.013320    2560 generic.go:356] "Generic (PLEG): container finished" podID="3632e237-bca4-47f1-afd8-06b62910ad82" containerID="40c01bba111039616587bb8c60339ff99f1be55b901746bd96b60e6c486647a3" exitCode=0
Jul 28 23:03:09.013755 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:09.013408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf" event={"ID":"3632e237-bca4-47f1-afd8-06b62910ad82","Type":"ContainerDied","Data":"40c01bba111039616587bb8c60339ff99f1be55b901746bd96b60e6c486647a3"}
Jul 28 23:03:09.152433 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:09.152408    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"
Jul 28 23:03:09.305183 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:09.305072    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3632e237-bca4-47f1-afd8-06b62910ad82-maas-api-service-ca\") pod \"3632e237-bca4-47f1-afd8-06b62910ad82\" (UID: \"3632e237-bca4-47f1-afd8-06b62910ad82\") "
Jul 28 23:03:09.305441 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:09.305419    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3632e237-bca4-47f1-afd8-06b62910ad82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3632e237-bca4-47f1-afd8-06b62910ad82" (UID: "3632e237-bca4-47f1-afd8-06b62910ad82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:03:09.406275 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:09.406242    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3632e237-bca4-47f1-afd8-06b62910ad82-maas-api-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:03:10.020140 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:10.020112    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"
Jul 28 23:03:10.020498 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:10.020143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf" event={"ID":"3632e237-bca4-47f1-afd8-06b62910ad82","Type":"ContainerDied","Data":"f57f3627739b3cf6ef5e542e40e621165420df29ce719c71adba6138deb5cf36"}
Jul 28 23:03:10.020498 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:10.020186    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f57f3627739b3cf6ef5e542e40e621165420df29ce719c71adba6138deb5cf36"
Jul 28 23:03:17.267308 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.267274    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"]
Jul 28 23:03:17.267934 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.267520    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd" podUID="ff793013-3b7b-4e94-9626-244418ec0d7c" containerName="maas-api" containerID="cri-o://077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf" gracePeriod=30
Jul 28 23:03:17.291424 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.291396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:17.296032 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.296005    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn"]
Jul 28 23:03:17.298661 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.298639    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-faf483-29754660-5s7kn"]
Jul 28 23:03:17.361828 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.361790    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:17.424349 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.424320    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:17.449814 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.449782    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"]
Jul 28 23:03:17.453155 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.453131    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-faf483-9dbkf"]
Jul 28 23:03:17.482510 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.482469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:17.536470 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.536443    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 23:03:17.575593 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.575562    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff793013-3b7b-4e94-9626-244418ec0d7c-maas-api-tls\") pod \"ff793013-3b7b-4e94-9626-244418ec0d7c\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") "
Jul 28 23:03:17.575789 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.575611    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff793013-3b7b-4e94-9626-244418ec0d7c-openshift-service-ca\") pod \"ff793013-3b7b-4e94-9626-244418ec0d7c\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") "
Jul 28 23:03:17.575789 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.575643    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xrn4\" (UniqueName: \"kubernetes.io/projected/ff793013-3b7b-4e94-9626-244418ec0d7c-kube-api-access-4xrn4\") pod \"ff793013-3b7b-4e94-9626-244418ec0d7c\" (UID: \"ff793013-3b7b-4e94-9626-244418ec0d7c\") "
Jul 28 23:03:17.576022 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.575995    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff793013-3b7b-4e94-9626-244418ec0d7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff793013-3b7b-4e94-9626-244418ec0d7c" (UID: "ff793013-3b7b-4e94-9626-244418ec0d7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:03:17.577691 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.577667    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff793013-3b7b-4e94-9626-244418ec0d7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff793013-3b7b-4e94-9626-244418ec0d7c" (UID: "ff793013-3b7b-4e94-9626-244418ec0d7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:03:17.577819 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.577724    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff793013-3b7b-4e94-9626-244418ec0d7c-kube-api-access-4xrn4" (OuterVolumeSpecName: "kube-api-access-4xrn4") pod "ff793013-3b7b-4e94-9626-244418ec0d7c" (UID: "ff793013-3b7b-4e94-9626-244418ec0d7c"). InnerVolumeSpecName "kube-api-access-4xrn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:03:17.677381 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.677340    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff793013-3b7b-4e94-9626-244418ec0d7c-maas-api-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:03:17.677381 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.677373    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff793013-3b7b-4e94-9626-244418ec0d7c-openshift-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:03:17.677381 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.677384    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xrn4\" (UniqueName: \"kubernetes.io/projected/ff793013-3b7b-4e94-9626-244418ec0d7c-kube-api-access-4xrn4\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Jul 28 23:03:17.823699 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:17.823610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:18.065120 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.065062    2560 generic.go:356] "Generic (PLEG): container finished" podID="ff793013-3b7b-4e94-9626-244418ec0d7c" containerID="077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf" exitCode=0
Jul 28 23:03:18.065284 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.065129    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"
Jul 28 23:03:18.065284 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.065158    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd" event={"ID":"ff793013-3b7b-4e94-9626-244418ec0d7c","Type":"ContainerDied","Data":"077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf"}
Jul 28 23:03:18.065284 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.065217    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd" event={"ID":"ff793013-3b7b-4e94-9626-244418ec0d7c","Type":"ContainerDied","Data":"bdc7adfe4b28d013a2e53401f1f276459432a1ff27449a059fe9052fff82263f"}
Jul 28 23:03:18.065284 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.065265    2560 scope.go:117] "RemoveContainer" containerID="077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf"
Jul 28 23:03:18.075277 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.075251    2560 scope.go:117] "RemoveContainer" containerID="077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf"
Jul 28 23:03:18.075504 ip-10-0-129-50 kubenswrapper[2560]: E0728 23:03:18.075487    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf\": container with ID starting with 077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf not found: ID does not exist" containerID="077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf"
Jul 28 23:03:18.075578 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.075511    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf"} err="failed to get container status \"077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf\": rpc error: code = NotFound desc = could not find container \"077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf\": container with ID starting with 077eedfed1fb2744a2d08433a4d3df67297eceff711c028ab5955d5a727c1ebf not found: ID does not exist"
Jul 28 23:03:18.087270 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.087243    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"]
Jul 28 23:03:18.090651 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.090632    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-faf483-6f8d456c4-rvnrd"]
Jul 28 23:03:18.408489 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.408408    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3632e237-bca4-47f1-afd8-06b62910ad82" path="/var/lib/kubelet/pods/3632e237-bca4-47f1-afd8-06b62910ad82/volumes"
Jul 28 23:03:18.408846 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.408697    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af6d0e1-cf7f-464e-9e97-e1b4529eee82" path="/var/lib/kubelet/pods/3af6d0e1-cf7f-464e-9e97-e1b4529eee82/volumes"
Jul 28 23:03:18.409008 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:18.408995    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff793013-3b7b-4e94-9626-244418ec0d7c" path="/var/lib/kubelet/pods/ff793013-3b7b-4e94-9626-244418ec0d7c/volumes"
Jul 28 23:03:23.654811 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:23.654775    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:23.726523 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:23.726486    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:23.911253 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:23.911169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:24.154714 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:24.154680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:30.524086 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:30.524059    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7ccc5769-7dfws_e1d80ea2-cd03-4d00-89b4-df2716812207/copy-manifests/0.log"
Jul 28 23:03:30.530896 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:30.530865    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7ccc5769-7dfws_e1d80ea2-cd03-4d00-89b4-df2716812207/manager/0.log"
Jul 28 23:03:31.242053 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:31.242018    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754630-qg49p_3d4a20b5-7d2f-4c75-ac6c-b9a41db04bfc/cleanup/0.log"
Jul 28 23:03:31.348903 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:31.348878    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754645-mfk7h_2ac4035b-a9a7-413e-8fd2-20027d12f074/cleanup/0.log"
Jul 28 23:03:31.459400 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:31.459362    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754660-ph6qb_b4db883e-ad1c-4449-97cd-c8fda4f50eb7/cleanup/0.log"
Jul 28 23:03:31.569429 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:31.569345    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wknnf_8fd46e63-d882-4945-bc82-cdce94366ed2/postgres/0.log"
Jul 28 23:03:33.177555 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:33.177527    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bt4hb_c7885d1c-f834-4a26-9e45-85e170caa048/kuadrant-console-plugin/0.log"
Jul 28 23:03:34.241805 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:34.241773    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4df5cb96-dl8mh_f4727687-705d-4799-a3dc-9a90cb2adf8d/kube-auth-proxy/0.log"
Jul 28 23:03:34.604862 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:34.604773    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9jqb6_14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8/payload-pre-processing/0.log"
Jul 28 23:03:34.723576 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:34.723542    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7g7wm_3e34ca30-b28c-4cca-8472-d23bcf1ddf51/payload-processing/0.log"
Jul 28 23:03:35.304388 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:35.304361    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-b6mrj_63fc61f1-aed9-4494-836e-9eabe0b6d6cd/main/0.log"
Jul 28 23:03:35.417900 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:35.417869    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-drxck_0577729c-705f-46c3-8f0d-dbaa92cd51f2/main/0.log"
Jul 28 23:03:41.430900 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.430868    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tzj6t/must-gather-sg5qs"]
Jul 28 23:03:41.431318 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.431260    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff793013-3b7b-4e94-9626-244418ec0d7c" containerName="maas-api"
Jul 28 23:03:41.431318 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.431272    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff793013-3b7b-4e94-9626-244418ec0d7c" containerName="maas-api"
Jul 28 23:03:41.431318 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.431304    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3632e237-bca4-47f1-afd8-06b62910ad82" containerName="revoke-keys"
Jul 28 23:03:41.431318 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.431310    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3632e237-bca4-47f1-afd8-06b62910ad82" containerName="revoke-keys"
Jul 28 23:03:41.431451 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.431369    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff793013-3b7b-4e94-9626-244418ec0d7c" containerName="maas-api"
Jul 28 23:03:41.431451 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.431378    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3632e237-bca4-47f1-afd8-06b62910ad82" containerName="revoke-keys"
Jul 28 23:03:41.434561 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.434539    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tzj6t/must-gather-sg5qs"
Jul 28 23:03:41.436583 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.436553    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:41.437258 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.437241    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tzj6t\"/\"kube-root-ca.crt\""
Jul 28 23:03:41.438218 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.438198    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tzj6t\"/\"openshift-service-ca.crt\""
Jul 28 23:03:41.438317 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.438225    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tzj6t\"/\"default-dockercfg-gslgj\""
Jul 28 23:03:41.442569 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.442323    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tzj6t/must-gather-sg5qs"]
Jul 28 23:03:41.496990 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.496954    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log" limit=1024
Jul 28 23:03:41.605255 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.605225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab-must-gather-output\") pod \"must-gather-sg5qs\" (UID: \"dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab\") " pod="openshift-must-gather-tzj6t/must-gather-sg5qs"
Jul 28 23:03:41.605434 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.605264    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdjbv\" (UniqueName: \"kubernetes.io/projected/dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab-kube-api-access-rdjbv\") pod \"must-gather-sg5qs\" (UID: \"dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab\") " pod="openshift-must-gather-tzj6t/must-gather-sg5qs"
Jul 28 23:03:41.706219 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.706141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjbv\" (UniqueName: \"kubernetes.io/projected/dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab-kube-api-access-rdjbv\") pod \"must-gather-sg5qs\" (UID: \"dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab\") " pod="openshift-must-gather-tzj6t/must-gather-sg5qs"
Jul 28 23:03:41.706385 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.706265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab-must-gather-output\") pod \"must-gather-sg5qs\" (UID: \"dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab\") " pod="openshift-must-gather-tzj6t/must-gather-sg5qs"
Jul 28 23:03:41.706590 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.706569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab-must-gather-output\") pod \"must-gather-sg5qs\" (UID: \"dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab\") " pod="openshift-must-gather-tzj6t/must-gather-sg5qs"
Jul 28 23:03:41.713681 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.713655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjbv\" (UniqueName: \"kubernetes.io/projected/dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab-kube-api-access-rdjbv\") pod \"must-gather-sg5qs\" (UID: \"dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab\") " pod="openshift-must-gather-tzj6t/must-gather-sg5qs"
Jul 28 23:03:41.745765 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.745740    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tzj6t/must-gather-sg5qs"
Jul 28 23:03:41.879218 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:41.879193    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tzj6t/must-gather-sg5qs"]
Jul 28 23:03:41.881396 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:03:41.881371    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd2d8a26_49ab_4d80_ac71_a9faec2cc4ab.slice/crio-4077851cac87e575ef896f7be202ee64496e1e5751d14960cb10d8e390751675 WatchSource:0}: Error finding container 4077851cac87e575ef896f7be202ee64496e1e5751d14960cb10d8e390751675: Status 404 returned error can't find the container with id 4077851cac87e575ef896f7be202ee64496e1e5751d14960cb10d8e390751675
Jul 28 23:03:42.197882 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:42.197848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tzj6t/must-gather-sg5qs" event={"ID":"dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab","Type":"ContainerStarted","Data":"4077851cac87e575ef896f7be202ee64496e1e5751d14960cb10d8e390751675"}
Jul 28 23:03:43.209731 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:43.209689    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tzj6t/must-gather-sg5qs" event={"ID":"dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab","Type":"ContainerStarted","Data":"00f36979fe1915a298fdd66e18879510f81ab7e1027bf355bf07d7b986847136"}
Jul 28 23:03:43.210263 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:43.209747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tzj6t/must-gather-sg5qs" event={"ID":"dd2d8a26-49ab-4d80-ac71-a9faec2cc4ab","Type":"ContainerStarted","Data":"d986b93c166410ebcb25946e4306dbce0f026df4367fed990fd7680a282dfe98"}
Jul 28 23:03:43.224694 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:43.224636    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tzj6t/must-gather-sg5qs" podStartSLOduration=1.524847678 podStartE2EDuration="2.224616093s" totalImagesPullingTime="699.768415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:03:43.22445499 +0000 UTC m=+3225.346802885" watchObservedRunningTime="2026-07-28 23:03:43.224616093 +0000 UTC m=+3225.346963987"
Jul 28 23:03:44.136388 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:44.136357    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4dpq8_94c1f799-00f0-4a61-b199-8db35aa6169c/global-pull-secret-syncer/0.log"
Jul 28 23:03:44.284765 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:44.284729    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6frnt_1b043760-c4fb-44dc-b4fe-b5fef68a1068/konnectivity-agent/0.log"
Jul 28 23:03:44.387441 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:44.387368    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-50.ec2.internal_ace7f39d4fac06f2546ed5091afdf977/haproxy/0.log"
Jul 28 23:03:50.387356 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:50.387318    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bt4hb_c7885d1c-f834-4a26-9e45-85e170caa048/kuadrant-console-plugin/0.log"
Jul 28 23:03:52.398152 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.398069    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8b3885b-8def-406e-805f-98264697d736/alertmanager/0.log"
Jul 28 23:03:52.428293 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.428260    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8b3885b-8def-406e-805f-98264697d736/config-reloader/0.log"
Jul 28 23:03:52.453488 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.453463    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8b3885b-8def-406e-805f-98264697d736/kube-rbac-proxy-web/0.log"
Jul 28 23:03:52.477497 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.477469    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8b3885b-8def-406e-805f-98264697d736/kube-rbac-proxy/0.log"
Jul 28 23:03:52.501554 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.501515    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8b3885b-8def-406e-805f-98264697d736/kube-rbac-proxy-metric/0.log"
Jul 28 23:03:52.527479 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.527449    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8b3885b-8def-406e-805f-98264697d736/prom-label-proxy/0.log"
Jul 28 23:03:52.556840 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.556753    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8b3885b-8def-406e-805f-98264697d736/init-config-reloader/0.log"
Jul 28 23:03:52.637305 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.637273    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-5sbml_f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3/kube-state-metrics/0.log"
Jul 28 23:03:52.658787 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.658760    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-5sbml_f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3/kube-rbac-proxy-main/0.log"
Jul 28 23:03:52.683897 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.683869    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-5sbml_f61e6d68-6b06-4f2d-b1c1-7f44bc4a5af3/kube-rbac-proxy-self/0.log"
Jul 28 23:03:52.716926 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.716899    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8577ff485f-gxjgx_93e993a2-bcb4-4ed4-8d2e-55898a337031/metrics-server/0.log"
Jul 28 23:03:52.752736 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.752706    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-nj8k8_2eba8e1b-809c-4c88-9248-79ff922c0312/monitoring-plugin/0.log"
Jul 28 23:03:52.857515 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.857444    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rkmgt_5a87fbe1-e0c9-421d-a157-b29d988cc06d/node-exporter/0.log"
Jul 28 23:03:52.879241 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.879212    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rkmgt_5a87fbe1-e0c9-421d-a157-b29d988cc06d/kube-rbac-proxy/0.log"
Jul 28 23:03:52.902472 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:52.902435    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rkmgt_5a87fbe1-e0c9-421d-a157-b29d988cc06d/init-textfile/0.log"
Jul 28 23:03:53.015723 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.015696    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-zvr7l_a3847606-a998-4e33-977d-981a3181029a/kube-rbac-proxy-main/0.log"
Jul 28 23:03:53.040294 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.040262    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-zvr7l_a3847606-a998-4e33-977d-981a3181029a/kube-rbac-proxy-self/0.log"
Jul 28 23:03:53.064411 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.064379    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-zvr7l_a3847606-a998-4e33-977d-981a3181029a/openshift-state-metrics/0.log"
Jul 28 23:03:53.098537 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.098497    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f100bf34-981b-4725-8a12-0cc83cad2040/prometheus/0.log"
Jul 28 23:03:53.122125 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.122013    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f100bf34-981b-4725-8a12-0cc83cad2040/config-reloader/0.log"
Jul 28 23:03:53.159319 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.159289    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f100bf34-981b-4725-8a12-0cc83cad2040/thanos-sidecar/0.log"
Jul 28 23:03:53.184307 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.184263    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f100bf34-981b-4725-8a12-0cc83cad2040/kube-rbac-proxy-web/0.log"
Jul 28 23:03:53.207871 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.207831    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f100bf34-981b-4725-8a12-0cc83cad2040/kube-rbac-proxy/0.log"
Jul 28 23:03:53.231224 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.231179    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f100bf34-981b-4725-8a12-0cc83cad2040/kube-rbac-proxy-thanos/0.log"
Jul 28 23:03:53.255763 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.255721    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f100bf34-981b-4725-8a12-0cc83cad2040/init-config-reloader/0.log"
Jul 28 23:03:53.289215 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.289184    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-c22pg_6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f/prometheus-operator/0.log"
Jul 28 23:03:53.313707 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.313679    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-c22pg_6ba2fadb-e0b8-4ddc-9fe0-f6fb1862be7f/kube-rbac-proxy/0.log"
Jul 28 23:03:53.338570 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.338530    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-ckk4w_a40d15d5-0327-4557-90b6-0179ffa1a0cd/prometheus-operator-admission-webhook/0.log"
Jul 28 23:03:53.371022 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.370965    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-694bf69f78-8rflg_070c4254-3b07-487c-a50f-c8b605e20f65/telemeter-client/0.log"
Jul 28 23:03:53.394251 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.394157    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-694bf69f78-8rflg_070c4254-3b07-487c-a50f-c8b605e20f65/reload/0.log"
Jul 28 23:03:53.417527 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.417498    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-694bf69f78-8rflg_070c4254-3b07-487c-a50f-c8b605e20f65/kube-rbac-proxy/0.log"
Jul 28 23:03:53.455183 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.455150    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76fd97f949-lxgh7_9bf7316a-83ed-486a-a247-9445f637e721/thanos-query/0.log"
Jul 28 23:03:53.479418 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.479379    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76fd97f949-lxgh7_9bf7316a-83ed-486a-a247-9445f637e721/kube-rbac-proxy-web/0.log"
Jul 28 23:03:53.503565 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.503525    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76fd97f949-lxgh7_9bf7316a-83ed-486a-a247-9445f637e721/kube-rbac-proxy/0.log"
Jul 28 23:03:53.526119 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.526055    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76fd97f949-lxgh7_9bf7316a-83ed-486a-a247-9445f637e721/prom-label-proxy/0.log"
Jul 28 23:03:53.548887 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.548856    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76fd97f949-lxgh7_9bf7316a-83ed-486a-a247-9445f637e721/kube-rbac-proxy-rules/0.log"
Jul 28 23:03:53.572343 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:53.572309    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76fd97f949-lxgh7_9bf7316a-83ed-486a-a247-9445f637e721/kube-rbac-proxy-metrics/0.log"
Jul 28 23:03:54.161216 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.161177    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"]
Jul 28 23:03:54.168162 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.168125    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.173742 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.173711    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"]
Jul 28 23:03:54.248334 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.248288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-podres\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.248528 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.248345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-proc\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.248528 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.248403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-lib-modules\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.248641 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.248525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spt2p\" (UniqueName: \"kubernetes.io/projected/aa6a9438-faf6-4cef-85ff-21393f2a2248-kube-api-access-spt2p\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.248641 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.248577    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-sys\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.349733 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.349691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spt2p\" (UniqueName: \"kubernetes.io/projected/aa6a9438-faf6-4cef-85ff-21393f2a2248-kube-api-access-spt2p\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.349916 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.349763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-sys\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.349916 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.349798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-podres\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.349916 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.349894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-sys\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.350061 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.349936    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-podres\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.350061 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.349948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-proc\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.350061 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.350007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-lib-modules\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.350213 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.350119    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-proc\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.350213 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.350188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa6a9438-faf6-4cef-85ff-21393f2a2248-lib-modules\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.357464 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.357428    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spt2p\" (UniqueName: \"kubernetes.io/projected/aa6a9438-faf6-4cef-85ff-21393f2a2248-kube-api-access-spt2p\") pod \"perf-node-gather-daemonset-7ljst\" (UID: \"aa6a9438-faf6-4cef-85ff-21393f2a2248\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.493706 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.493666    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:54.685592 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:54.684856    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"]
Jul 28 23:03:54.691887 ip-10-0-129-50 kubenswrapper[2560]: W0728 23:03:54.691847    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaa6a9438_faf6_4cef_85ff_21393f2a2248.slice/crio-868df4042718bbc1dc07bc0c1c6f61a4f4b93034d5aadeeee77097b931267354 WatchSource:0}: Error finding container 868df4042718bbc1dc07bc0c1c6f61a4f4b93034d5aadeeee77097b931267354: Status 404 returned error can't find the container with id 868df4042718bbc1dc07bc0c1c6f61a4f4b93034d5aadeeee77097b931267354
Jul 28 23:03:55.308057 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:55.308020    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst" event={"ID":"aa6a9438-faf6-4cef-85ff-21393f2a2248","Type":"ContainerStarted","Data":"f69120c35cc037ba8bbe99c42a91bc0af5cda45d2e62ed8034d61fdc46b553fa"}
Jul 28 23:03:55.308261 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:55.308131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst" event={"ID":"aa6a9438-faf6-4cef-85ff-21393f2a2248","Type":"ContainerStarted","Data":"868df4042718bbc1dc07bc0c1c6f61a4f4b93034d5aadeeee77097b931267354"}
Jul 28 23:03:55.308261 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:55.308164    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:03:55.325297 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:55.325219    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst" podStartSLOduration=1.325197468 podStartE2EDuration="1.325197468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:03:55.325181482 +0000 UTC m=+3237.447529376" watchObservedRunningTime="2026-07-28 23:03:55.325197468 +0000 UTC m=+3237.447545357"
Jul 28 23:03:55.815225 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:55.815174    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66c9496995-kx5qc_ed8d3d3c-963e-455b-a6b5-7a2f59c998a4/console/0.log"
Jul 28 23:03:57.422168 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:57.422138    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5rhj2_752619e7-d2bb-4aa9-a2c9-fdbf24a1b643/dns/0.log"
Jul 28 23:03:57.447231 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:57.447185    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5rhj2_752619e7-d2bb-4aa9-a2c9-fdbf24a1b643/kube-rbac-proxy/0.log"
Jul 28 23:03:57.562913 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:57.562877    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4prbw_ba035fdb-cc39-4bfa-aa6b-81d417c51ad1/dns-node-resolver/0.log"
Jul 28 23:03:58.245322 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:58.245283    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x75qb_389f391a-1877-421e-932f-2dae44bf6738/node-ca/0.log"
Jul 28 23:03:59.362257 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:59.362226    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4df5cb96-dl8mh_f4727687-705d-4799-a3dc-9a90cb2adf8d/kube-auth-proxy/0.log"
Jul 28 23:03:59.498912 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:59.498878    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9jqb6_14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8/payload-pre-processing/0.log"
Jul 28 23:03:59.541479 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:03:59.541449    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7g7wm_3e34ca30-b28c-4cca-8472-d23bcf1ddf51/payload-processing/0.log"
Jul 28 23:04:00.426696 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:00.426668    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlm5v_80f95a34-965e-4d1f-907c-142af4243ca8/serve-healthcheck-canary/0.log"
Jul 28 23:04:00.917299 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:00.917270    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8tzhz_2e3f7284-9265-42d4-be4b-a570241a0313/kube-rbac-proxy/0.log"
Jul 28 23:04:00.940156 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:00.940124    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8tzhz_2e3f7284-9265-42d4-be4b-a570241a0313/exporter/0.log"
Jul 28 23:04:00.963247 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:00.963217    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8tzhz_2e3f7284-9265-42d4-be4b-a570241a0313/extractor/0.log"
Jul 28 23:04:01.327513 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:01.327487    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-7ljst"
Jul 28 23:04:04.220397 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:04.220362    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7ccc5769-7dfws_e1d80ea2-cd03-4d00-89b4-df2716812207/manager/0.log"
Jul 28 23:04:04.249822 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:04.249797    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7ccc5769-7dfws_e1d80ea2-cd03-4d00-89b4-df2716812207/copy-manifests/0.log"
Jul 28 23:04:04.891173 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:04.891142    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4df5cb96-dl8mh_f4727687-705d-4799-a3dc-9a90cb2adf8d/kube-auth-proxy/0.log"
Jul 28 23:04:04.979865 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:04.979834    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9jqb6_14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8/payload-pre-processing/0.log"
Jul 28 23:04:05.004972 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:05.004944    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7g7wm_3e34ca30-b28c-4cca-8472-d23bcf1ddf51/payload-processing/0.log"
Jul 28 23:04:05.765534 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:05.765499    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8586db4f6f-q5dd2_47f07dff-d56c-4cf8-a247-0f1c34a28170/manager/0.log"
Jul 28 23:04:05.826473 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:05.826442    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-974zq_6a69e702-574e-4375-96c4-e88faa1d12cb/openshift-lws-operator/0.log"
Jul 28 23:04:06.626029 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:06.625959    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4df5cb96-dl8mh_f4727687-705d-4799-a3dc-9a90cb2adf8d/kube-auth-proxy/0.log"
Jul 28 23:04:06.712122 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:06.712080    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-9jqb6_14e0eb05-7d4b-4f4b-8c01-5b671e6d1fb8/payload-pre-processing/0.log"
Jul 28 23:04:06.746690 ip-10-0-129-50 kubenswrapper[2560]: I0728 23:04:06.746664    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7g7wm_3e34ca30-b28c-4cca-8472-d23bcf1ddf51/payload-processing/0.log"