Jul 27 19:37:54.342992 ip-10-0-141-6 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 27 19:37:54.343004 ip-10-0-141-6 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 27 19:37:54.343014 ip-10-0-141-6 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 27 19:37:54.343325 ip-10-0-141-6 systemd[1]: Failed to start Kubernetes Kubelet. Jul 27 19:38:03.922426 ip-10-0-141-6 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 27 19:38:03.922440 ip-10-0-141-6 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 07a42416ae0f4c4d9ff90e608e6decda -- Jul 27 19:39:55.276643 ip-10-0-141-6 systemd[1]: Starting Kubernetes Kubelet... Jul 27 19:39:55.644966 ip-10-0-141-6 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 19:39:55.644966 ip-10-0-141-6 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 19:39:55.644966 ip-10-0-141-6 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 19:39:55.644966 ip-10-0-141-6 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 19:39:55.644966 ip-10-0-141-6 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 19:39:55.646000 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.645885 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 19:39:55.652038 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652017 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:39:55.652038 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652034 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:39:55.652038 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652038 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:39:55.652038 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652042 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:39:55.652038 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652045 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:39:55.652038 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652048 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652051 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652054 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652058 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652061 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652064 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652066 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652070 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652073 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652075 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652080 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652085 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652090 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652093 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652096 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652099 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652102 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652105 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652108 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652110 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:39:55.652266 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652113 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652115 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652119 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652121 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652124 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652127 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652130 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652133 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652135 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652138 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652141 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652145 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652148 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652150 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652153 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652156 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652159 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652162 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652164 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652167 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:39:55.652765 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652169 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652172 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652176 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652180 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652183 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652187 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652189 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652192 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652195 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652198 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652201 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652203 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652206 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652209 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652213 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652216 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652220 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652222 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652225 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652228 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:39:55.653241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652231 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652234 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652236 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652239 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652242 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652245 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652248 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652251 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652254 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652256 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652261 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652263 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652267 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652270 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652272 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652275 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652278 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652281 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652284 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:39:55.653741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652286 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652289 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652307 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652312 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652316 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652318 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652321 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652325 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652327 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652330 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652333 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652335 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652338 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652341 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.652343 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653068 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653075 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653079 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653082 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653084 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:39:55.654233 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653087 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653090 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653092 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653095 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653097 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653101 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653103 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653106 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653108 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653111 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653113 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653116 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653119 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653121 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653124 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653126 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653129 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653131 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653134 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653137 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:39:55.654741 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653139 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653142 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653145 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653147 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653150 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653152 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653155 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653157 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653160 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653162 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653164 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653167 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653169 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653172 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653174 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653178 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653182 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653184 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653187 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653189 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:39:55.655236 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653192 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653195 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653198 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653200 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653203 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653206 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653209 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653212 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653214 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653217 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653219 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653222 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653225 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653227 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653230 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653232 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653235 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653237 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653239 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:39:55.655750 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653242 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653246 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653250 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653253 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653257 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653260 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653263 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653265 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653268 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653270 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653273 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653275 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653278 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653280 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653283 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653285 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653288 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653290 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653314 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:39:55.656207 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653318 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653321 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653323 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653326 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653328 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653331 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653334 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653336 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653339 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653341 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653345 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653349 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653352 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653355 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653358 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653361 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653753 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653760 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653764 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653767 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:39:55.656687 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653770 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653773 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653775 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653778 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653780 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653783 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653786 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653788 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653791 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653793 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653796 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653798 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653801 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653806 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653809 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653812 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653815 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653819 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653822 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653824 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:39:55.657168 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653828 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653830 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653833 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653836 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653840 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653843 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653846 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653849 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653852 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653855 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653857 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653860 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653862 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653865 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653867 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653870 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653872 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653874 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653877 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653880 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:39:55.657668 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653882 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653886 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653889 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653892 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653894 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653897 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653899 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653902 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653905 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653910 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653913 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653915 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653918 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653921 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653923 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653926 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653928 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653931 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653933 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:39:55.658141 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653936 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653938 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653941 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653944 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653946 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653949 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653951 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653954 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653956 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653959 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653961 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653964 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653966 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653969 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653971 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653974 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653976 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653979 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653982 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:39:55.658623 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653985 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653987 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653990 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653994 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653996 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.653999 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654001 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654004 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654006 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654009 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654011 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654014 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654016 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654018 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654021 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654024 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654027 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654096 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654103 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654109 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654113 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 19:39:55.659113 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654117 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654121 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654125 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654129 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654132 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654135 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654138 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654141 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654144 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654147 2573 flags.go:64] FLAG: --cgroup-root="" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654150 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654153 2573 flags.go:64] FLAG: --client-ca-file="" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654157 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654160 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654165 2573 flags.go:64] FLAG: --cluster-domain="" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654168 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654171 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654174 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654177 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654181 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654184 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654187 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654190 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654192 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 27 19:39:55.659666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654195 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654198 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654203 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654206 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654210 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654213 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654216 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654219 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654222 2573 flags.go:64] FLAG: --enable-server="true" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654225 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654229 2573 flags.go:64] FLAG: --event-burst="100" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654232 2573 flags.go:64] FLAG: --event-qps="50" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654235 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654238 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654241 2573 flags.go:64] FLAG: --eviction-hard="" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654245 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654248 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654251 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654254 2573 flags.go:64] FLAG: --eviction-soft="" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654257 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654260 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654264 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654267 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654270 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654273 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 19:39:55.660228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654276 2573 flags.go:64] FLAG: --feature-gates="" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654280 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654283 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654286 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654289 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654308 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654312 2573 flags.go:64] FLAG: --help="false" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654315 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-141-6.ec2.internal" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654318 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654322 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654325 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654328 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654332 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654334 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654337 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654340 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654343 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654346 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654348 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654351 2573 flags.go:64] FLAG: --kube-reserved="" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654354 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654357 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654360 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654362 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 19:39:55.660871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654365 2573 flags.go:64] FLAG: --lock-file="" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654368 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654371 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654374 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654380 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654384 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654387 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654389 2573 flags.go:64] FLAG: --logging-format="text" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654392 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654395 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654398 2573 flags.go:64] FLAG: --manifest-url="" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654401 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654405 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654407 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654412 2573 flags.go:64] FLAG: --max-pods="110" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654414 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654417 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654421 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654424 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654427 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654430 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654433 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654441 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654444 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654447 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 19:39:55.661463 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654450 2573 flags.go:64] FLAG: --pod-cidr="" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654452 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654457 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654460 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654463 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654466 2573 flags.go:64] FLAG: --port="10250" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654469 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654472 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04fef8fc781db7a45" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654475 2573 flags.go:64] FLAG: --qos-reserved="" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654478 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654481 2573 flags.go:64] FLAG: --register-node="true" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654483 2573 flags.go:64] FLAG: --register-with-taints="" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654492 2573 flags.go:64] FLAG: --registry-burst="10" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654495 2573 flags.go:64] FLAG: --registry-qps="5" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654507 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654510 2573 flags.go:64] FLAG: --reserved-memory="" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654513 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654516 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654519 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654522 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654525 2573 flags.go:64] FLAG: --runonce="false" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654528 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654530 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654533 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654537 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 19:39:55.662099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654540 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654543 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654545 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654548 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654551 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654554 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654556 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654559 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654562 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654565 2573 flags.go:64] FLAG: --system-cgroups="" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654568 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654573 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654576 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654578 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654583 2573 flags.go:64] FLAG: --tls-min-version="" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654586 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654589 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654592 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654595 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654597 2573 flags.go:64] FLAG: --v="2" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654601 2573 flags.go:64] FLAG: --version="false" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654606 2573 flags.go:64] FLAG: --vmodule="" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654609 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.654612 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654748 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:39:55.662755 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654753 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654756 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654759 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654762 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654764 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654767 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654771 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654773 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654776 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654778 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654780 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654783 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654786 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654788 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654791 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654795 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654797 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654800 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654802 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654805 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:39:55.663419 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654808 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654810 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654813 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654815 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654818 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654820 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654823 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654825 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654827 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654830 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654833 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654836 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654838 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654841 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654843 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654846 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654848 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654851 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654855 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654857 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:39:55.663901 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654861 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654864 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654867 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654870 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654872 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654876 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654879 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654883 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654886 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654889 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654892 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654894 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654898 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654902 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654905 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654908 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654910 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654913 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654916 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:39:55.664399 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654918 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654920 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654923 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654926 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654929 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654931 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654934 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654936 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654939 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654941 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654944 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654949 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654951 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654954 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654956 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654959 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654961 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654963 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654966 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654968 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:39:55.664861 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654974 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654976 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654979 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654981 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654984 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654986 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654989 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654991 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654993 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654996 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.654998 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.655001 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.655004 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.655006 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.655009 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.655011 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.655014 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.655016 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:39:55.665383 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.655019 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.655626 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.663114 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.663131 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.663825 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.663832 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663891 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663896 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663899 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663902 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663906 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663909 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663912 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663915 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663919 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:39:55.665813 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663922 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663925 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663927 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663930 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663933 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663935 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663938 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663940 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663944 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663946 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663949 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663952 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663954 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663957 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663960 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663963 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663966 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663968 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663971 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:39:55.666194 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663976 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663980 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663983 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663987 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663991 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663994 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663997 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.663999 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664002 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664005 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664008 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664010 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664013 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664015 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664018 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664021 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664023 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664026 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664028 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:39:55.666680 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664031 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664033 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664036 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664038 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664041 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664043 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664045 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664048 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664050 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664052 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664055 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664058 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664060 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664064 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664067 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664070 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664072 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664075 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664077 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664079 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:39:55.667128 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664082 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664085 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664088 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664090 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664092 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664095 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664097 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664100 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664102 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664104 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664108 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664111 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664114 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664116 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664119 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664121 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664124 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664126 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664129 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664131 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:39:55.667633 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664134 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664136 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664139 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664142 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664145 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664148 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664151 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664153 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664156 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664158 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664161 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664163 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.664168 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664279 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664283 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:39:55.668111 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664286 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664289 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664311 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664315 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664318 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664320 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664323 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664326 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664329 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664332 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664334 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664337 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664340 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664342 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664346 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664348 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664351 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664354 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664356 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664359 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:39:55.668482 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664361 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664365 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664368 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664370 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664373 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664376 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664378 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664380 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664383 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664385 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664388 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664390 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664393 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664395 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664398 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664400 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664403 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664406 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664409 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:39:55.668947 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664411 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664414 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664417 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664419 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664422 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664424 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664426 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664429 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664431 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664434 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664436 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664439 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664441 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664443 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664446 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664449 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664451 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664454 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664457 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664459 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:39:55.669416 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664462 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664466 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664469 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664472 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664474 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664477 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664479 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664482 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664484 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664486 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664489 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664491 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664494 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664496 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664499 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664501 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664503 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664506 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664508 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:39:55.669902 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664512 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664515 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664517 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664520 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664523 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664525 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664528 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664531 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664534 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664537 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664540 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664544 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664546 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664549 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664551 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664554 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664556 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664558 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:39:55.670401 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:55.664561 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:39:55.670846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.664566 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 19:39:55.670846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.664695 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 19:39:55.670846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.669819 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 19:39:55.671357 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.671334 2573 server.go:1019] "Starting client certificate rotation" Jul 27 19:39:55.671464 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.671446 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 19:39:55.671523 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.671513 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 19:39:55.674509 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.674488 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 19:39:55.678000 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.677980 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 19:39:55.687928 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.687906 2573 log.go:25] "Validated CRI v1 runtime API" Jul 27 19:39:55.689960 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.689941 2573 log.go:25] "Validated CRI v1 image API" Jul 27 19:39:55.691515 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.691501 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 19:39:55.696566 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.696545 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dd9efc3e-87d3-4a27-8fa4-0501b4dc9987:/dev/nvme0n1p4 feecde51-e33f-40d3-865e-77dbe0b368c9:/dev/nvme0n1p3] Jul 27 19:39:55.696640 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.696564 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 19:39:55.700278 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.700259 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 19:39:55.702606 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.702482 2573 manager.go:216] Machine: {Timestamp:2026-07-27 19:39:55.700368688 +0000 UTC m=+0.312953274 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100630 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f40c7dce29117dc1f8672de4af448 SystemUUID:ec2f40c7-dce2-9117-dc1f-8672de4af448 BootID:07a42416-ae0f-4c4d-9ff9-0e608e6decda Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a9:bc:05:7a:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a9:bc:05:7a:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:a8:2b:92:e4:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 19:39:55.702606 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.702602 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 19:39:55.702708 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.702694 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 19:39:55.705792 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.705751 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 19:39:55.705937 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.705796 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 19:39:55.706605 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.706593 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 19:39:55.706639 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.706607 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 19:39:55.706639 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.706616 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 19:39:55.707431 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.707420 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 19:39:55.708330 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.708320 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 19:39:55.711006 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.710994 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 27 19:39:55.711174 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.711164 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 19:39:55.711213 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.711195 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 27 19:39:55.711213 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.711204 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 19:39:55.711281 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.711217 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 19:39:55.711281 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.711225 2573 kubelet.go:398] "Adding apiserver pod source" Jul 27 19:39:55.711281 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.711235 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 19:39:55.711682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.711669 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 19:39:55.711722 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.711692 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 19:39:55.713965 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.713946 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 19:39:55.715783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.715769 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 19:39:55.715848 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.715799 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 19:39:55.717320 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717307 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 19:39:55.717358 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717327 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 19:39:55.717358 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717333 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 19:39:55.717358 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717340 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 19:39:55.717358 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717348 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 19:39:55.717358 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717353 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 19:39:55.717502 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717361 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 19:39:55.717502 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717366 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 19:39:55.717502 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717373 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 19:39:55.717502 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717378 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 19:39:55.717502 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717387 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 19:39:55.717502 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.717392 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 19:39:55.718204 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.718192 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 19:39:55.718204 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.718203 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 19:39:55.721625 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.721611 2573 server.go:1268] "Started kubelet" Jul 27 19:39:55.721730 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.721704 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 19:39:55.721783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.721732 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 19:39:55.721783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.721748 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 19:39:55.722436 ip-10-0-141-6 systemd[1]: Started Kubernetes Kubelet. Jul 27 19:39:55.724585 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.724562 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 19:39:55.725458 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.725430 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 19:39:55.725458 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.725434 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 19:39:55.725651 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.725494 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-6.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 19:39:55.726591 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.726563 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 27 19:39:55.728235 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.728215 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z2rsq" Jul 27 19:39:55.729447 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.729426 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 19:39:55.733350 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.732514 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-6.ec2.internal.18c63cd6d6b12fdc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-6.ec2.internal,UID:ip-10-0-141-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-6.ec2.internal,},FirstTimestamp:2026-07-27 19:39:55.721584604 +0000 UTC m=+0.334169178,LastTimestamp:2026-07-27 19:39:55.721584604 +0000 UTC m=+0.334169178,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-6.ec2.internal,}" Jul 27 19:39:55.733962 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.733936 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 19:39:55.733962 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.733941 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 19:39:55.734320 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.734260 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:55.734883 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.734861 2573 factory.go:55] Registering systemd factory Jul 27 19:39:55.734883 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.734881 2573 factory.go:223] Registration of the systemd container factory successfully Jul 27 19:39:55.735030 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735016 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 19:39:55.735082 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735035 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 19:39:55.735195 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735180 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 27 19:39:55.735195 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735194 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 27 19:39:55.735322 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735228 2573 factory.go:153] Registering CRI-O factory Jul 27 19:39:55.735322 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735247 2573 factory.go:223] Registration of the crio container factory successfully Jul 27 19:39:55.735322 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735321 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 19:39:55.735438 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735347 2573 factory.go:103] Registering Raw factory Jul 27 19:39:55.735438 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735362 2573 manager.go:1196] Started watching for new ooms in manager Jul 27 19:39:55.735600 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.735582 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 19:39:55.736473 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.736459 2573 manager.go:349] Starting recovery of all containers Jul 27 19:39:55.737465 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.737445 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z2rsq" Jul 27 19:39:55.738249 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.738229 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 19:39:55.738336 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.738271 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 19:39:55.748148 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.748128 2573 manager.go:354] Recovery completed Jul 27 19:39:55.752963 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.752947 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:39:55.755065 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.755047 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:39:55.755114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.755079 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:39:55.755114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.755090 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:39:55.755689 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.755674 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 19:39:55.755689 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.755686 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 19:39:55.755814 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.755703 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 27 19:39:55.755814 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.755757 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:39:55.756589 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.756572 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:39:55.756666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.756604 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:39:55.756666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.756614 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:39:55.756821 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.756735 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-6.ec2.internal.18c63cd6d8b011bb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-6.ec2.internal,UID:ip-10-0-141-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-6.ec2.internal,},FirstTimestamp:2026-07-27 19:39:55.755065787 +0000 UTC m=+0.367650361,LastTimestamp:2026-07-27 19:39:55.755065787 +0000 UTC m=+0.367650361,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-6.ec2.internal,}" Jul 27 19:39:55.758243 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.758230 2573 policy_none.go:49] "None policy: Start" Jul 27 19:39:55.758330 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.758249 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 19:39:55.758330 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.758263 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 19:39:55.760272 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.760259 2573 policy_none.go:47] "Start" Jul 27 19:39:55.800040 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.799851 2573 manager.go:337] "Starting Device Plugin manager" Jul 27 19:39:55.800125 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.800071 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 19:39:55.800125 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.800086 2573 server.go:87] "Starting device plugin registration server" Jul 27 19:39:55.800376 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.800364 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 19:39:55.800411 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.800380 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 19:39:55.800505 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.800487 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 19:39:55.800593 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.800581 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 19:39:55.800645 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.800595 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 19:39:55.801090 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.801067 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 19:39:55.801170 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.801124 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:55.867528 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.867483 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 19:39:55.869657 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.868880 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 19:39:55.869657 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.868918 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 19:39:55.869657 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.868941 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 19:39:55.869657 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.868974 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 19:39:55.872125 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.872107 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:39:55.900521 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.900457 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:39:55.901349 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.901334 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:39:55.901423 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.901365 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:39:55.901423 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.901375 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:39:55.901423 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.901399 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-6.ec2.internal" Jul 27 19:39:55.912323 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.912289 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-6.ec2.internal" Jul 27 19:39:55.912386 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.912325 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-6.ec2.internal\": node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:55.952960 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.952929 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:55.970084 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.970059 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal"] Jul 27 19:39:55.970150 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.970124 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:39:55.970987 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.970971 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:39:55.971072 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.971002 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:39:55.971072 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.971040 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:39:55.973354 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.973342 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:39:55.973484 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.973471 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" Jul 27 19:39:55.973521 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.973501 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:39:55.974077 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.974050 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:39:55.974164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.974085 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:39:55.974164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.974095 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:39:55.974164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.974051 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:39:55.974164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.974160 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:39:55.974332 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.974172 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:39:55.976247 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.976232 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" Jul 27 19:39:55.976341 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.976256 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:39:55.976881 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.976866 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:39:55.976951 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.976893 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:39:55.976951 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:55.976903 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:39:55.990833 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.990815 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-6.ec2.internal\" not found" node="ip-10-0-141-6.ec2.internal" Jul 27 19:39:55.994903 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:55.994884 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-6.ec2.internal\" not found" node="ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.053775 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.053751 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:56.136942 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.136912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a05271190b4f7ec0b7de75b83b48e465-config\") pod \"kube-apiserver-proxy-ip-10-0-141-6.ec2.internal\" (UID: \"a05271190b4f7ec0b7de75b83b48e465\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.137072 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.136948 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49126f0e72cc82b2517d88af4afacf27-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"49126f0e72cc82b2517d88af4afacf27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.137072 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.136983 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49126f0e72cc82b2517d88af4afacf27-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"49126f0e72cc82b2517d88af4afacf27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.154736 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.154683 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:56.237421 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.237392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a05271190b4f7ec0b7de75b83b48e465-config\") pod \"kube-apiserver-proxy-ip-10-0-141-6.ec2.internal\" (UID: \"a05271190b4f7ec0b7de75b83b48e465\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.237421 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.237421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49126f0e72cc82b2517d88af4afacf27-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"49126f0e72cc82b2517d88af4afacf27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.237540 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.237438 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49126f0e72cc82b2517d88af4afacf27-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"49126f0e72cc82b2517d88af4afacf27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.237540 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.237500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49126f0e72cc82b2517d88af4afacf27-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"49126f0e72cc82b2517d88af4afacf27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.237540 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.237510 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49126f0e72cc82b2517d88af4afacf27-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal\" (UID: \"49126f0e72cc82b2517d88af4afacf27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.237631 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.237510 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a05271190b4f7ec0b7de75b83b48e465-config\") pod \"kube-apiserver-proxy-ip-10-0-141-6.ec2.internal\" (UID: \"a05271190b4f7ec0b7de75b83b48e465\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.255139 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.255107 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:56.293328 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.293284 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.299157 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.299141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.355479 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.355443 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:56.456059 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.456027 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:56.556528 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.556491 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-6.ec2.internal\" not found" Jul 27 19:39:56.581614 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.581594 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:39:56.623089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.623063 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:39:56.634594 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.634577 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.649501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.649488 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 19:39:56.650404 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.650392 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" Jul 27 19:39:56.659152 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.659137 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 19:39:56.670277 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.670256 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 19:39:56.670386 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.670363 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 19:39:56.670482 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.670394 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 19:39:56.670482 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.670413 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 19:39:56.711543 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.711485 2573 apiserver.go:52] "Watching apiserver" Jul 27 19:39:56.730308 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.729819 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 19:39:56.731391 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.731364 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-wm7nj","openshift-multus/network-metrics-daemon-q2r5h","openshift-network-diagnostics/network-check-target-md4vm","openshift-network-operator/iptables-alerter-c845d","openshift-ovn-kubernetes/ovnkube-node-7fg5b","kube-system/konnectivity-agent-fs6t5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86","openshift-cluster-node-tuning-operator/tuned-nrl7w","openshift-dns/node-resolver-55zm9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal","openshift-multus/multus-bdx6w","kube-system/global-pull-secret-syncer-69k4m","kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal","openshift-image-registry/node-ca-wmbgb"] Jul 27 19:39:56.733741 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.733724 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.734023 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.734008 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 19:39:56.735725 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.735709 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.737388 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.737376 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 19:39:56.737626 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.737615 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:39:56.737676 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.737661 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:39:56.737989 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.737978 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 19:39:56.738119 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.738102 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hl6mg\"" Jul 27 19:39:56.739557 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.739545 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.739776 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.739747 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 19:39:56.739776 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.739770 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 19:39:56.739969 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.739956 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sxrpj\"" Jul 27 19:39:56.740037 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740001 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 19:39:56.740037 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740010 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 19:39:56.740189 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-daemon-config\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740234 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-cnibin\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740234 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-device-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.740234 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.740380 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-system-cni-dir\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740380 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-var-lib-cni-bin\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740380 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-var-lib-cni-multus\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740380 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740310 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-etc-kubernetes\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740380 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:39:56.740380 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-sys-fs\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.740380 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cqw\" (UniqueName: \"kubernetes.io/projected/52ef6606-39e7-4c70-8386-086a93bf5393-kube-api-access-n9cqw\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.740380 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740364 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-socket-dir-parent\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rvt\" (UniqueName: \"kubernetes.io/projected/312ad691-dc6a-4105-9390-b32089a8ca6d-kube-api-access-p5rvt\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740450 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-registration-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-cni-dir\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740493 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-var-lib-kubelet\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-run-multus-certs\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-socket-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-os-release\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/312ad691-dc6a-4105-9390-b32089a8ca6d-cni-binary-copy\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740644 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-run-k8s-cni-cncf-io\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740665 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-run-netns\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740685 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-hostroot\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.740714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.740716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-conf-dir\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.741667 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.741652 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.743675 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.743660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:39:56.743834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.743795 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 19:34:55 +0000 UTC" deadline="2028-01-02 11:47:02.908764048 +0000 UTC" Jul 27 19:39:56.743917 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.743835 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12568h7m6.164931558s" Jul 27 19:39:56.744200 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.744188 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 19:39:56.744671 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.744657 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mphh9\"" Jul 27 19:39:56.744842 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.744831 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 19:39:56.745241 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.745229 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 19:39:56.745679 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.745666 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.745755 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.745742 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 19:39:56.746211 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.746193 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:39:56.747524 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.747509 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 19:39:56.747836 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.747824 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.749914 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.749900 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:39:56.749977 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.749960 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:39:56.750691 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.750676 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 19:39:56.750760 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.750699 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tqzq6\"" Jul 27 19:39:56.750760 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.750734 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 19:39:56.751244 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.751231 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l7tx2\"" Jul 27 19:39:56.751913 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.751892 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 19:39:56.751913 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.751904 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fn5xb\"" Jul 27 19:39:56.752049 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.751918 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 19:39:56.752049 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.751891 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 19:39:56.752049 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.751988 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x9fcl\"" Jul 27 19:39:56.752049 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.751918 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 19:39:56.752049 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.752013 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 19:39:56.752049 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.751897 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 19:39:56.752350 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.752172 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 19:39:56.752350 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.752177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.752350 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.752327 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 19:39:56.752457 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.752330 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 19:39:56.754134 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.754120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.755156 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.755140 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:39:56.755211 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.755186 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 19:39:56.755248 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.755188 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cqw8x\"" Jul 27 19:39:56.756089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.756077 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:56.756135 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.756124 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:39:56.757609 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.757597 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 19:39:56.757769 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.757755 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 19:39:56.758010 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.757997 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b7t4v\"" Jul 27 19:39:56.760005 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.759987 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 19:39:56.810207 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.810186 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7hkn4" Jul 27 19:39:56.819861 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.819845 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7hkn4" Jul 27 19:39:56.837146 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.837131 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 19:39:56.841828 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.841807 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.841957 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.841840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cf1c32f-1dc1-45e1-8992-6443f5ea0df1-serviceca\") pod \"node-ca-wmbgb\" (UID: \"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1\") " pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.841957 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.841865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-run-multus-certs\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.841957 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.841887 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-os-release\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.841957 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.841908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7782c74-246e-4a12-9312-b84a6e64667a-ovnkube-config\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.841957 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.841928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-sys\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.841957 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.841929 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-run-multus-certs\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.841957 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.841947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-host\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-socket-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-etc-kubernetes\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-slash\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-cni-bin\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087c69da-4dc1-4b01-9b0b-f972323a4124-tmp-dir\") pod \"node-resolver-55zm9\" (UID: \"087c69da-4dc1-4b01-9b0b-f972323a4124\") " pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842110 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-socket-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cf1c32f-1dc1-45e1-8992-6443f5ea0df1-host\") pod \"node-ca-wmbgb\" (UID: \"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1\") " pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-etc-kubernetes\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zmz\" (UniqueName: \"kubernetes.io/projected/d461523b-81bb-4af6-a2cf-1c77074efbb6-kube-api-access-75zmz\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rvt\" (UniqueName: \"kubernetes.io/projected/312ad691-dc6a-4105-9390-b32089a8ca6d-kube-api-access-p5rvt\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.842261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1873aafc-ad68-4fc5-9c04-f6070959451b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842321 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-etc-openvswitch\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842337 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1873aafc-ad68-4fc5-9c04-f6070959451b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1873aafc-ad68-4fc5-9c04-f6070959451b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/087c69da-4dc1-4b01-9b0b-f972323a4124-hosts-file\") pod \"node-resolver-55zm9\" (UID: \"087c69da-4dc1-4b01-9b0b-f972323a4124\") " pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-run-netns\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-conf-dir\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842461 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25d9\" (UniqueName: \"kubernetes.io/projected/2fa58f40-2a4b-4dfb-80e0-0a701161e32c-kube-api-access-l25d9\") pod \"iptables-alerter-c845d\" (UID: \"2fa58f40-2a4b-4dfb-80e0-0a701161e32c\") " pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-run-netns\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-run-netns\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842507 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-conf-dir\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-run-systemd\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7782c74-246e-4a12-9312-b84a6e64667a-ovnkube-script-lib\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-run\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-system-cni-dir\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.842706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842635 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-var-lib-cni-multus\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-system-cni-dir\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842683 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-var-lib-cni-multus\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-var-lib-openvswitch\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-run-openvswitch\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-node-log\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7782c74-246e-4a12-9312-b84a6e64667a-env-overrides\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842805 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-sysctl-d\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cqw\" (UniqueName: \"kubernetes.io/projected/52ef6606-39e7-4c70-8386-086a93bf5393-kube-api-access-n9cqw\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-kubelet\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7782c74-246e-4a12-9312-b84a6e64667a-ovn-node-metrics-cert\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-kubernetes\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29722d54-c0d2-4dea-9355-34ceb072e8bf-tmp\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txd8v\" (UniqueName: \"kubernetes.io/projected/29722d54-c0d2-4dea-9355-34ceb072e8bf-kube-api-access-txd8v\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-registration-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.842995 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-var-lib-kubelet\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.843198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-modprobe-d\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-daemon-config\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843046 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-var-lib-kubelet\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-registration-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843063 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2fa58f40-2a4b-4dfb-80e0-0a701161e32c-iptables-alerter-script\") pod \"iptables-alerter-c845d\" (UID: \"2fa58f40-2a4b-4dfb-80e0-0a701161e32c\") " pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh5d8\" (UniqueName: \"kubernetes.io/projected/5cf1c32f-1dc1-45e1-8992-6443f5ea0df1-kube-api-access-xh5d8\") pod \"node-ca-wmbgb\" (UID: \"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1\") " pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843115 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-sysctl-conf\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-lib-modules\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-var-lib-kubelet\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-cnibin\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-cnibin\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843330 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-cnibin\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843340 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-systemd-units\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-run-ovn\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.843846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4cfed4dd-9be6-4717-889d-7813b1dceb33-konnectivity-ca\") pod \"konnectivity-agent-fs6t5\" (UID: \"4cfed4dd-9be6-4717-889d-7813b1dceb33\") " pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-tuned\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-socket-dir-parent\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843501 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-daemon-config\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843544 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-socket-dir-parent\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9911a42-c569-4d6b-89e9-08a1551fd7cc-dbus\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-log-socket\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-cni-netd\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-cni-dir\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843650 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-system-cni-dir\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9911a42-c569-4d6b-89e9-08a1551fd7cc-kubelet-config\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843690 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-systemd\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-os-release\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843758 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-multus-cni-dir\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843771 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/312ad691-dc6a-4105-9390-b32089a8ca6d-cni-binary-copy\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.844501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843796 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-run-k8s-cni-cncf-io\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843800 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-os-release\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843818 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-hostroot\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843829 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-run-k8s-cni-cncf-io\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843844 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmms\" (UniqueName: \"kubernetes.io/projected/1873aafc-ad68-4fc5-9c04-f6070959451b-kube-api-access-dbmms\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843850 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-hostroot\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fa58f40-2a4b-4dfb-80e0-0a701161e32c-host-slash\") pod \"iptables-alerter-c845d\" (UID: \"2fa58f40-2a4b-4dfb-80e0-0a701161e32c\") " pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdbdv\" (UniqueName: \"kubernetes.io/projected/087c69da-4dc1-4b01-9b0b-f972323a4124-kube-api-access-xdbdv\") pod \"node-resolver-55zm9\" (UID: \"087c69da-4dc1-4b01-9b0b-f972323a4124\") " pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843917 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-device-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843930 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-var-lib-cni-bin\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843967 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-device-dir\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.843981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5kql\" (UniqueName: \"kubernetes.io/projected/a7782c74-246e-4a12-9312-b84a6e64667a-kube-api-access-w5kql\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.844007 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/312ad691-dc6a-4105-9390-b32089a8ca6d-host-var-lib-cni-bin\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.844021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4cfed4dd-9be6-4717-889d-7813b1dceb33-agent-certs\") pod \"konnectivity-agent-fs6t5\" (UID: \"4cfed4dd-9be6-4717-889d-7813b1dceb33\") " pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.844052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-sysconfig\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.844077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-sys-fs\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.844100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:56.845089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.844207 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52ef6606-39e7-4c70-8386-086a93bf5393-sys-fs\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.845559 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.844652 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/312ad691-dc6a-4105-9390-b32089a8ca6d-cni-binary-copy\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.851931 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.851916 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 19:39:56.852773 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.852755 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:39:56.852848 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.852776 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:39:56.852848 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.852787 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2nhpw for pod openshift-network-diagnostics/network-check-target-md4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:39:56.852848 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.852842 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw podName:f736b83c-af3d-47fb-a7d2-e3349cd84e30 nodeName:}" failed. No retries permitted until 2026-07-27 19:39:57.352824715 +0000 UTC m=+1.965409287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2nhpw" (UniqueName: "kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw") pod "network-check-target-md4vm" (UID: "f736b83c-af3d-47fb-a7d2-e3349cd84e30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:39:56.854648 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.854633 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rvt\" (UniqueName: \"kubernetes.io/projected/312ad691-dc6a-4105-9390-b32089a8ca6d-kube-api-access-p5rvt\") pod \"multus-bdx6w\" (UID: \"312ad691-dc6a-4105-9390-b32089a8ca6d\") " pod="openshift-multus/multus-bdx6w" Jul 27 19:39:56.854840 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.854825 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cqw\" (UniqueName: \"kubernetes.io/projected/52ef6606-39e7-4c70-8386-086a93bf5393-kube-api-access-n9cqw\") pod \"aws-ebs-csi-driver-node-fcf86\" (UID: \"52ef6606-39e7-4c70-8386-086a93bf5393\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25d9\" (UniqueName: \"kubernetes.io/projected/2fa58f40-2a4b-4dfb-80e0-0a701161e32c-kube-api-access-l25d9\") pod \"iptables-alerter-c845d\" (UID: \"2fa58f40-2a4b-4dfb-80e0-0a701161e32c\") " pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-run-netns\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-run-systemd\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7782c74-246e-4a12-9312-b84a6e64667a-ovnkube-script-lib\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945594 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-run\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-var-lib-openvswitch\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-run-openvswitch\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-node-log\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7782c74-246e-4a12-9312-b84a6e64667a-env-overrides\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-sysctl-d\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-kubelet\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7782c74-246e-4a12-9312-b84a6e64667a-ovn-node-metrics-cert\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945828 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-kubernetes\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29722d54-c0d2-4dea-9355-34ceb072e8bf-tmp\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txd8v\" (UniqueName: \"kubernetes.io/projected/29722d54-c0d2-4dea-9355-34ceb072e8bf-kube-api-access-txd8v\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-modprobe-d\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.947169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.945985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2fa58f40-2a4b-4dfb-80e0-0a701161e32c-iptables-alerter-script\") pod \"iptables-alerter-c845d\" (UID: \"2fa58f40-2a4b-4dfb-80e0-0a701161e32c\") " pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946011 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh5d8\" (UniqueName: \"kubernetes.io/projected/5cf1c32f-1dc1-45e1-8992-6443f5ea0df1-kube-api-access-xh5d8\") pod \"node-ca-wmbgb\" (UID: \"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1\") " pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946089 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-sysctl-conf\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-lib-modules\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946164 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-var-lib-kubelet\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-cnibin\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-systemd-units\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-run-ovn\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4cfed4dd-9be6-4717-889d-7813b1dceb33-konnectivity-ca\") pod \"konnectivity-agent-fs6t5\" (UID: \"4cfed4dd-9be6-4717-889d-7813b1dceb33\") " pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-tuned\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9911a42-c569-4d6b-89e9-08a1551fd7cc-dbus\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-log-socket\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-cni-netd\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-system-cni-dir\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9911a42-c569-4d6b-89e9-08a1551fd7cc-kubelet-config\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:56.948036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-systemd\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946774 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmms\" (UniqueName: \"kubernetes.io/projected/1873aafc-ad68-4fc5-9c04-f6070959451b-kube-api-access-dbmms\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fa58f40-2a4b-4dfb-80e0-0a701161e32c-host-slash\") pod \"iptables-alerter-c845d\" (UID: \"2fa58f40-2a4b-4dfb-80e0-0a701161e32c\") " pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdbdv\" (UniqueName: \"kubernetes.io/projected/087c69da-4dc1-4b01-9b0b-f972323a4124-kube-api-access-xdbdv\") pod \"node-resolver-55zm9\" (UID: \"087c69da-4dc1-4b01-9b0b-f972323a4124\") " pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946928 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5kql\" (UniqueName: \"kubernetes.io/projected/a7782c74-246e-4a12-9312-b84a6e64667a-kube-api-access-w5kql\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4cfed4dd-9be6-4717-889d-7813b1dceb33-agent-certs\") pod \"konnectivity-agent-fs6t5\" (UID: \"4cfed4dd-9be6-4717-889d-7813b1dceb33\") " pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.946983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-sysconfig\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cf1c32f-1dc1-45e1-8992-6443f5ea0df1-serviceca\") pod \"node-ca-wmbgb\" (UID: \"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1\") " pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-os-release\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947200 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-cnibin\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947492 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-run-systemd\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947512 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-modprobe-d\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-run-netns\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947836 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-node-log\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947838 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2fa58f40-2a4b-4dfb-80e0-0a701161e32c-iptables-alerter-script\") pod \"iptables-alerter-c845d\" (UID: \"2fa58f40-2a4b-4dfb-80e0-0a701161e32c\") " pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.947940 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-kubelet\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.948846 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.947985 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948027 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-run-openvswitch\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.948060 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret podName:f9911a42-c569-4d6b-89e9-08a1551fd7cc nodeName:}" failed. No retries permitted until 2026-07-27 19:39:57.448043871 +0000 UTC m=+2.060628451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret") pod "global-pull-secret-syncer-69k4m" (UID: "f9911a42-c569-4d6b-89e9-08a1551fd7cc") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948113 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-kubernetes\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948326 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7782c74-246e-4a12-9312-b84a6e64667a-ovnkube-script-lib\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948353 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-systemd\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948334 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948430 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fa58f40-2a4b-4dfb-80e0-0a701161e32c-host-slash\") pod \"iptables-alerter-c845d\" (UID: \"2fa58f40-2a4b-4dfb-80e0-0a701161e32c\") " pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.948466 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948492 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-sysctl-d\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:56.948536 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs podName:d461523b-81bb-4af6-a2cf-1c77074efbb6 nodeName:}" failed. No retries permitted until 2026-07-27 19:39:57.448521713 +0000 UTC m=+2.061106287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs") pod "network-metrics-daemon-q2r5h" (UID: "d461523b-81bb-4af6-a2cf-1c77074efbb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948599 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9911a42-c569-4d6b-89e9-08a1551fd7cc-kubelet-config\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948642 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-cni-netd\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-var-lib-kubelet\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948697 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9911a42-c569-4d6b-89e9-08a1551fd7cc-dbus\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948740 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.949643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948892 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-run\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-lib-modules\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-log-socket\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949046 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-var-lib-openvswitch\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7782c74-246e-4a12-9312-b84a6e64667a-env-overrides\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-systemd-units\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949165 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-system-cni-dir\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.948440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7782c74-246e-4a12-9312-b84a6e64667a-ovnkube-config\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949220 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-run-ovn\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-sys\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-host\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-slash\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949387 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-cni-bin\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087c69da-4dc1-4b01-9b0b-f972323a4124-tmp-dir\") pod \"node-resolver-55zm9\" (UID: \"087c69da-4dc1-4b01-9b0b-f972323a4124\") " pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cf1c32f-1dc1-45e1-8992-6443f5ea0df1-host\") pod \"node-ca-wmbgb\" (UID: \"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1\") " pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949460 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1873aafc-ad68-4fc5-9c04-f6070959451b-os-release\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949463 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-sysctl-conf\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949524 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cf1c32f-1dc1-45e1-8992-6443f5ea0df1-host\") pod \"node-ca-wmbgb\" (UID: \"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1\") " pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.951245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949531 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-sysconfig\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zmz\" (UniqueName: \"kubernetes.io/projected/d461523b-81bb-4af6-a2cf-1c77074efbb6-kube-api-access-75zmz\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1873aafc-ad68-4fc5-9c04-f6070959451b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-etc-openvswitch\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1873aafc-ad68-4fc5-9c04-f6070959451b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1873aafc-ad68-4fc5-9c04-f6070959451b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/087c69da-4dc1-4b01-9b0b-f972323a4124-hosts-file\") pod \"node-resolver-55zm9\" (UID: \"087c69da-4dc1-4b01-9b0b-f972323a4124\") " pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087c69da-4dc1-4b01-9b0b-f972323a4124-tmp-dir\") pod \"node-resolver-55zm9\" (UID: \"087c69da-4dc1-4b01-9b0b-f972323a4124\") " pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949932 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/087c69da-4dc1-4b01-9b0b-f972323a4124-hosts-file\") pod \"node-resolver-55zm9\" (UID: \"087c69da-4dc1-4b01-9b0b-f972323a4124\") " pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-host\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949614 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29722d54-c0d2-4dea-9355-34ceb072e8bf-sys\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949637 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-slash\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.949648 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-cni-bin\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.950042 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.950050 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4cfed4dd-9be6-4717-889d-7813b1dceb33-konnectivity-ca\") pod \"konnectivity-agent-fs6t5\" (UID: \"4cfed4dd-9be6-4717-889d-7813b1dceb33\") " pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.950096 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7782c74-246e-4a12-9312-b84a6e64667a-ovn-node-metrics-cert\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.951834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.950139 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7782c74-246e-4a12-9312-b84a6e64667a-etc-openvswitch\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.952280 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.950500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cf1c32f-1dc1-45e1-8992-6443f5ea0df1-serviceca\") pod \"node-ca-wmbgb\" (UID: \"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1\") " pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.952280 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.950505 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29722d54-c0d2-4dea-9355-34ceb072e8bf-tmp\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.952280 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.950604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1873aafc-ad68-4fc5-9c04-f6070959451b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.952280 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.950603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1873aafc-ad68-4fc5-9c04-f6070959451b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.952280 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.951050 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29722d54-c0d2-4dea-9355-34ceb072e8bf-etc-tuned\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.952280 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.951160 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7782c74-246e-4a12-9312-b84a6e64667a-ovnkube-config\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.952280 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.950701 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1873aafc-ad68-4fc5-9c04-f6070959451b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.952280 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.952204 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4cfed4dd-9be6-4717-889d-7813b1dceb33-agent-certs\") pod \"konnectivity-agent-fs6t5\" (UID: \"4cfed4dd-9be6-4717-889d-7813b1dceb33\") " pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:39:56.958932 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.958916 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdbdv\" (UniqueName: \"kubernetes.io/projected/087c69da-4dc1-4b01-9b0b-f972323a4124-kube-api-access-xdbdv\") pod \"node-resolver-55zm9\" (UID: \"087c69da-4dc1-4b01-9b0b-f972323a4124\") " pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:56.965915 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.965868 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5kql\" (UniqueName: \"kubernetes.io/projected/a7782c74-246e-4a12-9312-b84a6e64667a-kube-api-access-w5kql\") pod \"ovnkube-node-7fg5b\" (UID: \"a7782c74-246e-4a12-9312-b84a6e64667a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:56.966181 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.966167 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh5d8\" (UniqueName: \"kubernetes.io/projected/5cf1c32f-1dc1-45e1-8992-6443f5ea0df1-kube-api-access-xh5d8\") pod \"node-ca-wmbgb\" (UID: \"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1\") " pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:56.967077 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.967056 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zmz\" (UniqueName: \"kubernetes.io/projected/d461523b-81bb-4af6-a2cf-1c77074efbb6-kube-api-access-75zmz\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:39:56.967154 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.967127 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmms\" (UniqueName: \"kubernetes.io/projected/1873aafc-ad68-4fc5-9c04-f6070959451b-kube-api-access-dbmms\") pod \"multus-additional-cni-plugins-wm7nj\" (UID: \"1873aafc-ad68-4fc5-9c04-f6070959451b\") " pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:56.968620 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.968598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txd8v\" (UniqueName: \"kubernetes.io/projected/29722d54-c0d2-4dea-9355-34ceb072e8bf-kube-api-access-txd8v\") pod \"tuned-nrl7w\" (UID: \"29722d54-c0d2-4dea-9355-34ceb072e8bf\") " pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:56.968759 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.968744 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25d9\" (UniqueName: \"kubernetes.io/projected/2fa58f40-2a4b-4dfb-80e0-0a701161e32c-kube-api-access-l25d9\") pod \"iptables-alerter-c845d\" (UID: \"2fa58f40-2a4b-4dfb-80e0-0a701161e32c\") " pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:56.970371 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:56.970351 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49126f0e72cc82b2517d88af4afacf27.slice/crio-183bd3906e65ecd22a7a1f8bcbebfdd707846dd478a451a4cda5e94f0a3567f1 WatchSource:0}: Error finding container 183bd3906e65ecd22a7a1f8bcbebfdd707846dd478a451a4cda5e94f0a3567f1: Status 404 returned error can't find the container with id 183bd3906e65ecd22a7a1f8bcbebfdd707846dd478a451a4cda5e94f0a3567f1 Jul 27 19:39:56.970753 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:56.970735 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05271190b4f7ec0b7de75b83b48e465.slice/crio-aeab21a460ca0475cd22acde3147ba73f103546d059c49dd5e242f4e1a98ce50 WatchSource:0}: Error finding container aeab21a460ca0475cd22acde3147ba73f103546d059c49dd5e242f4e1a98ce50: Status 404 returned error can't find the container with id aeab21a460ca0475cd22acde3147ba73f103546d059c49dd5e242f4e1a98ce50 Jul 27 19:39:56.975262 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:56.975248 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 19:39:57.064394 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.064357 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" Jul 27 19:39:57.070316 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.070278 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bdx6w" Jul 27 19:39:57.073402 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:57.073383 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ef6606_39e7_4c70_8386_086a93bf5393.slice/crio-e8898f48c7d3a188b8550076945c4c24ac3340eeec77c2c36d2bf2645fcea7f9 WatchSource:0}: Error finding container e8898f48c7d3a188b8550076945c4c24ac3340eeec77c2c36d2bf2645fcea7f9: Status 404 returned error can't find the container with id e8898f48c7d3a188b8550076945c4c24ac3340eeec77c2c36d2bf2645fcea7f9 Jul 27 19:39:57.079379 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:57.079358 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312ad691_dc6a_4105_9390_b32089a8ca6d.slice/crio-c61b34c3e4fb0a5373fe6afd30999277763ad8b6b72d5e0ec15ba90a6d1bfee2 WatchSource:0}: Error finding container c61b34c3e4fb0a5373fe6afd30999277763ad8b6b72d5e0ec15ba90a6d1bfee2: Status 404 returned error can't find the container with id c61b34c3e4fb0a5373fe6afd30999277763ad8b6b72d5e0ec15ba90a6d1bfee2 Jul 27 19:39:57.084900 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.084864 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c845d" Jul 27 19:39:57.093119 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:57.093101 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa58f40_2a4b_4dfb_80e0_0a701161e32c.slice/crio-0972d98697e4a37a9af4f4b42050c711fe91804a948f2841b20e6d97311cd504 WatchSource:0}: Error finding container 0972d98697e4a37a9af4f4b42050c711fe91804a948f2841b20e6d97311cd504: Status 404 returned error can't find the container with id 0972d98697e4a37a9af4f4b42050c711fe91804a948f2841b20e6d97311cd504 Jul 27 19:39:57.099578 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.099563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:39:57.104227 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.104211 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:39:57.108555 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:57.108537 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7782c74_246e_4a12_9312_b84a6e64667a.slice/crio-6d61a666ff18b43dadb6a0e51ee5b5e0f011ab02e55b27ee9bba1597c49dc34c WatchSource:0}: Error finding container 6d61a666ff18b43dadb6a0e51ee5b5e0f011ab02e55b27ee9bba1597c49dc34c: Status 404 returned error can't find the container with id 6d61a666ff18b43dadb6a0e51ee5b5e0f011ab02e55b27ee9bba1597c49dc34c Jul 27 19:39:57.113197 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:57.113180 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfed4dd_9be6_4717_889d_7813b1dceb33.slice/crio-bfd40bbf0e34cc227b8a04c8f4ca30328e67e6ba1278af70eb387726823b435c WatchSource:0}: Error finding container bfd40bbf0e34cc227b8a04c8f4ca30328e67e6ba1278af70eb387726823b435c: Status 404 returned error can't find the container with id bfd40bbf0e34cc227b8a04c8f4ca30328e67e6ba1278af70eb387726823b435c Jul 27 19:39:57.136570 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.136544 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:39:57.147703 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.147685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-55zm9" Jul 27 19:39:57.156302 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:57.156267 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087c69da_4dc1_4b01_9b0b_f972323a4124.slice/crio-748c7fd15d4f18296b6c501ba5a2c8cd61897ab2bd1dc944584b048a0fa89c06 WatchSource:0}: Error finding container 748c7fd15d4f18296b6c501ba5a2c8cd61897ab2bd1dc944584b048a0fa89c06: Status 404 returned error can't find the container with id 748c7fd15d4f18296b6c501ba5a2c8cd61897ab2bd1dc944584b048a0fa89c06 Jul 27 19:39:57.164967 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.164952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wmbgb" Jul 27 19:39:57.172522 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:57.172504 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf1c32f_1dc1_45e1_8992_6443f5ea0df1.slice/crio-b4210bf1690da9161af77fe1fd4670548626094c7a33f447430081d3d899cb54 WatchSource:0}: Error finding container b4210bf1690da9161af77fe1fd4670548626094c7a33f447430081d3d899cb54: Status 404 returned error can't find the container with id b4210bf1690da9161af77fe1fd4670548626094c7a33f447430081d3d899cb54 Jul 27 19:39:57.182523 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.182504 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" Jul 27 19:39:57.188022 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.188004 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" Jul 27 19:39:57.189541 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:57.189523 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29722d54_c0d2_4dea_9355_34ceb072e8bf.slice/crio-ee3f7c47bed6cbd0dad31d2aa07ba007dd3eafe36a9fbf5b5b3b1da86a1a422c WatchSource:0}: Error finding container ee3f7c47bed6cbd0dad31d2aa07ba007dd3eafe36a9fbf5b5b3b1da86a1a422c: Status 404 returned error can't find the container with id ee3f7c47bed6cbd0dad31d2aa07ba007dd3eafe36a9fbf5b5b3b1da86a1a422c Jul 27 19:39:57.196150 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:39:57.196136 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1873aafc_ad68_4fc5_9c04_f6070959451b.slice/crio-a88359219afb1afce89e2df864861f6e1e7ab13d42152288c4b5be54653b58fd WatchSource:0}: Error finding container a88359219afb1afce89e2df864861f6e1e7ab13d42152288c4b5be54653b58fd: Status 404 returned error can't find the container with id a88359219afb1afce89e2df864861f6e1e7ab13d42152288c4b5be54653b58fd Jul 27 19:39:57.453800 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.453769 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:57.453947 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.453830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:39:57.453947 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.453854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:39:57.454060 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:57.453983 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:39:57.454060 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:57.454001 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:39:57.454060 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:57.454014 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2nhpw for pod openshift-network-diagnostics/network-check-target-md4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:39:57.454182 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:57.454066 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw podName:f736b83c-af3d-47fb-a7d2-e3349cd84e30 nodeName:}" failed. No retries permitted until 2026-07-27 19:39:58.454049266 +0000 UTC m=+3.066633838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nhpw" (UniqueName: "kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw") pod "network-check-target-md4vm" (UID: "f736b83c-af3d-47fb-a7d2-e3349cd84e30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:39:57.454459 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:57.454436 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:39:57.454613 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:57.454470 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:39:57.454613 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:57.454497 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs podName:d461523b-81bb-4af6-a2cf-1c77074efbb6 nodeName:}" failed. No retries permitted until 2026-07-27 19:39:58.454480738 +0000 UTC m=+3.067065306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs") pod "network-metrics-daemon-q2r5h" (UID: "d461523b-81bb-4af6-a2cf-1c77074efbb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:39:57.454613 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:57.454520 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret podName:f9911a42-c569-4d6b-89e9-08a1551fd7cc nodeName:}" failed. No retries permitted until 2026-07-27 19:39:58.454505641 +0000 UTC m=+3.067090251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret") pod "global-pull-secret-syncer-69k4m" (UID: "f9911a42-c569-4d6b-89e9-08a1551fd7cc") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:39:57.551812 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.551784 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:39:57.822941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.822852 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:34:56 +0000 UTC" deadline="2028-03-15 15:51:15.185864266 +0000 UTC" Jul 27 19:39:57.822941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.822897 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14324h11m17.362973543s" Jul 27 19:39:57.872189 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.872123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:39:57.872387 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:57.872247 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:39:57.898406 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.898351 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fs6t5" event={"ID":"4cfed4dd-9be6-4717-889d-7813b1dceb33","Type":"ContainerStarted","Data":"bfd40bbf0e34cc227b8a04c8f4ca30328e67e6ba1278af70eb387726823b435c"} Jul 27 19:39:57.915060 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.910375 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" event={"ID":"a7782c74-246e-4a12-9312-b84a6e64667a","Type":"ContainerStarted","Data":"6d61a666ff18b43dadb6a0e51ee5b5e0f011ab02e55b27ee9bba1597c49dc34c"} Jul 27 19:39:57.915060 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.913537 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" event={"ID":"52ef6606-39e7-4c70-8386-086a93bf5393","Type":"ContainerStarted","Data":"e8898f48c7d3a188b8550076945c4c24ac3340eeec77c2c36d2bf2645fcea7f9"} Jul 27 19:39:57.931688 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.931654 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" event={"ID":"49126f0e72cc82b2517d88af4afacf27","Type":"ContainerStarted","Data":"183bd3906e65ecd22a7a1f8bcbebfdd707846dd478a451a4cda5e94f0a3567f1"} Jul 27 19:39:57.943943 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.943910 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" event={"ID":"a05271190b4f7ec0b7de75b83b48e465","Type":"ContainerStarted","Data":"aeab21a460ca0475cd22acde3147ba73f103546d059c49dd5e242f4e1a98ce50"} Jul 27 19:39:57.958675 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.958638 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" event={"ID":"29722d54-c0d2-4dea-9355-34ceb072e8bf","Type":"ContainerStarted","Data":"ee3f7c47bed6cbd0dad31d2aa07ba007dd3eafe36a9fbf5b5b3b1da86a1a422c"} Jul 27 19:39:57.970116 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.970079 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c845d" event={"ID":"2fa58f40-2a4b-4dfb-80e0-0a701161e32c","Type":"ContainerStarted","Data":"0972d98697e4a37a9af4f4b42050c711fe91804a948f2841b20e6d97311cd504"} Jul 27 19:39:57.992571 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:57.992535 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bdx6w" event={"ID":"312ad691-dc6a-4105-9390-b32089a8ca6d","Type":"ContainerStarted","Data":"c61b34c3e4fb0a5373fe6afd30999277763ad8b6b72d5e0ec15ba90a6d1bfee2"} Jul 27 19:39:58.004905 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.004866 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerStarted","Data":"a88359219afb1afce89e2df864861f6e1e7ab13d42152288c4b5be54653b58fd"} Jul 27 19:39:58.026179 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.026097 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wmbgb" event={"ID":"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1","Type":"ContainerStarted","Data":"b4210bf1690da9161af77fe1fd4670548626094c7a33f447430081d3d899cb54"} Jul 27 19:39:58.035286 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.035193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-55zm9" event={"ID":"087c69da-4dc1-4b01-9b0b-f972323a4124","Type":"ContainerStarted","Data":"748c7fd15d4f18296b6c501ba5a2c8cd61897ab2bd1dc944584b048a0fa89c06"} Jul 27 19:39:58.460975 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.460933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:39:58.461147 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.460987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:39:58.461147 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.461039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:58.464541 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.464509 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:39:58.464541 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.464523 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:39:58.464711 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.464555 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:39:58.464711 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.464575 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2nhpw for pod openshift-network-diagnostics/network-check-target-md4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:39:58.464711 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.464597 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs podName:d461523b-81bb-4af6-a2cf-1c77074efbb6 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:00.464578313 +0000 UTC m=+5.077162882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs") pod "network-metrics-daemon-q2r5h" (UID: "d461523b-81bb-4af6-a2cf-1c77074efbb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:39:58.464711 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.464630 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw podName:f736b83c-af3d-47fb-a7d2-e3349cd84e30 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:00.464609828 +0000 UTC m=+5.077194398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nhpw" (UniqueName: "kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw") pod "network-check-target-md4vm" (UID: "f736b83c-af3d-47fb-a7d2-e3349cd84e30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:39:58.464898 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.464800 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:39:58.464898 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.464844 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret podName:f9911a42-c569-4d6b-89e9-08a1551fd7cc nodeName:}" failed. No retries permitted until 2026-07-27 19:40:00.464833089 +0000 UTC m=+5.077417653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret") pod "global-pull-secret-syncer-69k4m" (UID: "f9911a42-c569-4d6b-89e9-08a1551fd7cc") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:39:58.660860 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.660639 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:39:58.823357 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.823206 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:34:56 +0000 UTC" deadline="2028-01-19 00:01:43.355716628 +0000 UTC" Jul 27 19:39:58.823357 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.823264 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12964h21m44.532456725s" Jul 27 19:39:58.869947 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.869919 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:39:58.870094 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.870033 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:39:58.870456 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.870433 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:39:58.870632 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:58.870598 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:39:58.942394 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:58.942096 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:39:59.871490 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:39:59.871457 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:39:59.871935 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:39:59.871648 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:00.480505 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:00.480473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:00.480505 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:00.480519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:00.480761 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:00.480564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:00.480761 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.480646 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:40:00.480761 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.480661 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:40:00.480761 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.480716 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret podName:f9911a42-c569-4d6b-89e9-08a1551fd7cc nodeName:}" failed. No retries permitted until 2026-07-27 19:40:04.480699487 +0000 UTC m=+9.093284062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret") pod "global-pull-secret-syncer-69k4m" (UID: "f9911a42-c569-4d6b-89e9-08a1551fd7cc") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:40:00.480761 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.480736 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs podName:d461523b-81bb-4af6-a2cf-1c77074efbb6 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:04.48072601 +0000 UTC m=+9.093310573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs") pod "network-metrics-daemon-q2r5h" (UID: "d461523b-81bb-4af6-a2cf-1c77074efbb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:40:00.480997 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.480784 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:40:00.480997 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.480799 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:40:00.480997 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.480812 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2nhpw for pod openshift-network-diagnostics/network-check-target-md4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:40:00.480997 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.480854 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw podName:f736b83c-af3d-47fb-a7d2-e3349cd84e30 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:04.480838656 +0000 UTC m=+9.093423221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nhpw" (UniqueName: "kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw") pod "network-check-target-md4vm" (UID: "f736b83c-af3d-47fb-a7d2-e3349cd84e30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:40:00.869805 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:00.869720 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:00.869979 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.869869 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:00.869979 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:00.869966 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:00.870076 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:00.870054 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:01.870360 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:01.870288 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:01.870825 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:01.870452 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:02.869902 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:02.869867 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:02.870111 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:02.869873 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:02.870111 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:02.870010 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:02.870111 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:02.870069 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:03.871510 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:03.871424 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:03.871944 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:03.871554 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:04.518355 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:04.517497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:04.518355 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:04.517573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:04.518355 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:04.517604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:04.518355 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.517762 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:40:04.518355 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.517780 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:40:04.518355 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.517793 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2nhpw for pod openshift-network-diagnostics/network-check-target-md4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:40:04.518355 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.517850 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw podName:f736b83c-af3d-47fb-a7d2-e3349cd84e30 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:12.517831761 +0000 UTC m=+17.130416325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nhpw" (UniqueName: "kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw") pod "network-check-target-md4vm" (UID: "f736b83c-af3d-47fb-a7d2-e3349cd84e30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:40:04.518355 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.518249 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:40:04.518355 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.518313 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret podName:f9911a42-c569-4d6b-89e9-08a1551fd7cc nodeName:}" failed. No retries permitted until 2026-07-27 19:40:12.518281287 +0000 UTC m=+17.130865853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret") pod "global-pull-secret-syncer-69k4m" (UID: "f9911a42-c569-4d6b-89e9-08a1551fd7cc") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:40:04.519040 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.518377 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:40:04.519040 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.518410 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs podName:d461523b-81bb-4af6-a2cf-1c77074efbb6 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:12.518397571 +0000 UTC m=+17.130982136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs") pod "network-metrics-daemon-q2r5h" (UID: "d461523b-81bb-4af6-a2cf-1c77074efbb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:40:04.869886 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:04.869812 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:04.869886 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:04.869850 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:04.870078 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.869966 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:04.870173 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:04.870112 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:05.871103 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:05.871052 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:05.871591 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:05.871156 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:06.870138 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:06.870097 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:06.870319 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:06.870243 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:06.870319 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:06.870311 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:06.870788 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:06.870750 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:07.870125 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:07.870087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:07.870664 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:07.870232 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:08.869364 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:08.869334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:08.869498 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:08.869470 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:08.869498 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:08.869467 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:08.869615 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:08.869556 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:09.869429 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:09.869390 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:09.869784 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:09.869500 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:10.869674 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:10.869643 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:10.870136 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:10.869774 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:10.870136 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:10.869826 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:10.870136 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:10.869932 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:11.869834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:11.869805 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:11.870234 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:11.869921 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:12.579823 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:12.579797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:12.579959 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:12.579854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:12.579959 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.579935 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:40:12.580062 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:12.579958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:12.580062 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.579983 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs podName:d461523b-81bb-4af6-a2cf-1c77074efbb6 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:28.579966507 +0000 UTC m=+33.192551067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs") pod "network-metrics-daemon-q2r5h" (UID: "d461523b-81bb-4af6-a2cf-1c77074efbb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:40:12.580062 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.579935 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:40:12.580062 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.580033 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:40:12.580062 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.580046 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:40:12.580062 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.580052 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret podName:f9911a42-c569-4d6b-89e9-08a1551fd7cc nodeName:}" failed. No retries permitted until 2026-07-27 19:40:28.580031588 +0000 UTC m=+33.192616150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret") pod "global-pull-secret-syncer-69k4m" (UID: "f9911a42-c569-4d6b-89e9-08a1551fd7cc") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:40:12.580062 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.580060 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2nhpw for pod openshift-network-diagnostics/network-check-target-md4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:40:12.580317 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.580099 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw podName:f736b83c-af3d-47fb-a7d2-e3349cd84e30 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:28.580086398 +0000 UTC m=+33.192670959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nhpw" (UniqueName: "kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw") pod "network-check-target-md4vm" (UID: "f736b83c-af3d-47fb-a7d2-e3349cd84e30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:40:12.869232 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:12.869156 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:12.869480 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:12.869165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:12.869480 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.869267 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:12.869480 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:12.869389 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:13.870129 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:13.870095 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:13.870587 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:13.870218 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:14.870134 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:14.870100 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:14.870614 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:14.870138 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:14.870614 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:14.870231 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:14.870614 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:14.870376 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:15.870672 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:15.870634 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:15.871091 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:15.870726 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:16.870133 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:16.870097 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:16.870320 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:16.870109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:16.870320 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:16.870209 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:16.870320 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:16.870265 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:17.869628 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:17.869174 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:17.870311 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:17.869737 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:18.083695 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.083667 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bdx6w" event={"ID":"312ad691-dc6a-4105-9390-b32089a8ca6d","Type":"ContainerStarted","Data":"8a138cd88feeb191c8e62edbc0ac2fc815000b21c2d2149cf8b47738e29d7a53"} Jul 27 19:40:18.085023 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.085002 2573 generic.go:356] "Generic (PLEG): container finished" podID="1873aafc-ad68-4fc5-9c04-f6070959451b" containerID="65dddd4fb4174055832e550240ee906e4ba7bac9b96ab106ac7467f93dca7955" exitCode=0 Jul 27 19:40:18.085111 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.085062 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerDied","Data":"65dddd4fb4174055832e550240ee906e4ba7bac9b96ab106ac7467f93dca7955"} Jul 27 19:40:18.086450 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.086427 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wmbgb" event={"ID":"5cf1c32f-1dc1-45e1-8992-6443f5ea0df1","Type":"ContainerStarted","Data":"2c53c57267ec2524f0c66cff7a27602762d41d34771d1025ec503dc5aef2f65d"} Jul 27 19:40:18.087936 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.087916 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-55zm9" event={"ID":"087c69da-4dc1-4b01-9b0b-f972323a4124","Type":"ContainerStarted","Data":"3b1bc97e08a5c4abcad902ee61db8e02d9b42f81d9b8194d045e3316e5d76d9a"} Jul 27 19:40:18.089212 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.089193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fs6t5" event={"ID":"4cfed4dd-9be6-4717-889d-7813b1dceb33","Type":"ContainerStarted","Data":"5c7940eb9df55ddc60835eb028bcde3741ce5af179a25690604eee102b28d401"} Jul 27 19:40:18.092664 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.092643 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" event={"ID":"a7782c74-246e-4a12-9312-b84a6e64667a","Type":"ContainerStarted","Data":"dd7f30b6c136874090e73a10b7a4b7707153740c149d85b58daa6a8be77e424d"} Jul 27 19:40:18.092763 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.092710 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" event={"ID":"a7782c74-246e-4a12-9312-b84a6e64667a","Type":"ContainerStarted","Data":"be68330710b2ce4072b6012b48c18eb7a54a917ddc8ec0829953c03aa1ff03bd"} Jul 27 19:40:18.092763 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.092757 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" event={"ID":"a7782c74-246e-4a12-9312-b84a6e64667a","Type":"ContainerStarted","Data":"bb83bf93804cbe4a8a1801293d4191728cc35f555f4b80cb0e4e86124006e58b"} Jul 27 19:40:18.092872 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.092809 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" event={"ID":"a7782c74-246e-4a12-9312-b84a6e64667a","Type":"ContainerStarted","Data":"b4f8e2921ebd3629fc386fc020a0f8db95f53e58bd5811524879df8af8ead24a"} Jul 27 19:40:18.092872 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.092856 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" event={"ID":"a7782c74-246e-4a12-9312-b84a6e64667a","Type":"ContainerStarted","Data":"7a87f0974fd6f616bade3fb2d75b2cec3b91263a003d150650789fb570849cbe"} Jul 27 19:40:18.094245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.094219 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" event={"ID":"52ef6606-39e7-4c70-8386-086a93bf5393","Type":"ContainerStarted","Data":"73a523359d7e898a957e66716ec157292140061d0e39727ce9fa18c50af43e68"} Jul 27 19:40:18.096908 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.096875 2573 generic.go:356] "Generic (PLEG): container finished" podID="49126f0e72cc82b2517d88af4afacf27" containerID="1901e78c9272bf67133a4611557bf7b45c6a476cf3f1760807eac27cd2559aa2" exitCode=0 Jul 27 19:40:18.097006 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.096957 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" event={"ID":"49126f0e72cc82b2517d88af4afacf27","Type":"ContainerDied","Data":"1901e78c9272bf67133a4611557bf7b45c6a476cf3f1760807eac27cd2559aa2"} Jul 27 19:40:18.098887 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.098864 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" event={"ID":"a05271190b4f7ec0b7de75b83b48e465","Type":"ContainerStarted","Data":"9a594875f0dbacef8f6f28caa2f71b8a2c6273a8945b6c0a6749574418e4f428"} Jul 27 19:40:18.100573 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.100548 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" event={"ID":"29722d54-c0d2-4dea-9355-34ceb072e8bf","Type":"ContainerStarted","Data":"12644cb9aab2f5e0f1bbcf9b2b5ef79857905f10fb42ac168b531f99148285d9"} Jul 27 19:40:18.125724 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.125671 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nrl7w" podStartSLOduration=3.374359346 podStartE2EDuration="23.125658304s" totalImagesPullingTime="19.751298958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:18.125310705 +0000 UTC m=+22.737895312" watchObservedRunningTime="2026-07-27 19:40:18.125658304 +0000 UTC m=+22.738242886" Jul 27 19:40:18.143586 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.143540 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bdx6w" podStartSLOduration=3.219370865 podStartE2EDuration="23.143525054s" totalImagesPullingTime="19.924154189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:18.143356448 +0000 UTC m=+22.755941061" watchObservedRunningTime="2026-07-27 19:40:18.143525054 +0000 UTC m=+22.756109639" Jul 27 19:40:18.159892 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.159487 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wmbgb" podStartSLOduration=3.445651961 podStartE2EDuration="23.159473212s" totalImagesPullingTime="19.713821251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:18.159161496 +0000 UTC m=+22.771746081" watchObservedRunningTime="2026-07-27 19:40:18.159473212 +0000 UTC m=+22.772057794" Jul 27 19:40:18.211643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.211598 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-6.ec2.internal" podStartSLOduration=22.211579889 podStartE2EDuration="22.211579889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:18.193576435 +0000 UTC m=+22.806161034" watchObservedRunningTime="2026-07-27 19:40:18.211579889 +0000 UTC m=+22.824164471" Jul 27 19:40:18.212032 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.212000 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fs6t5" podStartSLOduration=3.438704244 podStartE2EDuration="23.211989738s" totalImagesPullingTime="19.773285494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:18.211178795 +0000 UTC m=+22.823763377" watchObservedRunningTime="2026-07-27 19:40:18.211989738 +0000 UTC m=+22.824574571" Jul 27 19:40:18.265478 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.265441 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:18.327177 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.327148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:18.428268 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.428212 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:18.858123 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.858095 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 19:40:18.869812 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.869793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:18.870111 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:18.869793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:18.870111 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:18.869889 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:18.870111 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:18.869964 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:19.107453 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.107363 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" event={"ID":"a7782c74-246e-4a12-9312-b84a6e64667a","Type":"ContainerStarted","Data":"23e4d9cec21bf0b2fb8c9d1ebee7218dc6e2679b0817a02d22f0557a73ee22b5"} Jul 27 19:40:19.109242 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.109205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" event={"ID":"52ef6606-39e7-4c70-8386-086a93bf5393","Type":"ContainerStarted","Data":"f1347471d2ff1a8268112e38e80ce208170d192531f65dadfda8e63a4c8b1ad9"} Jul 27 19:40:19.111372 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.111340 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" event={"ID":"49126f0e72cc82b2517d88af4afacf27","Type":"ContainerStarted","Data":"0f3e8b7fc5875422db1af0e6ba1d45c510081a8e542e9a14e2eda4c6cabca4e9"} Jul 27 19:40:19.112848 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.112822 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c845d" event={"ID":"2fa58f40-2a4b-4dfb-80e0-0a701161e32c","Type":"ContainerStarted","Data":"0b5074ee1482ffc9fe3fde383e7a7b00e554f81f8a0a0b19f08893025887fd6d"} Jul 27 19:40:19.128221 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.128174 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-55zm9" podStartSLOduration=4.280908622 podStartE2EDuration="24.128157709s" totalImagesPullingTime="19.847249087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:18.231392993 +0000 UTC m=+22.843977598" watchObservedRunningTime="2026-07-27 19:40:19.128157709 +0000 UTC m=+23.740742293" Jul 27 19:40:19.128410 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.128255 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-6.ec2.internal" podStartSLOduration=23.128247283 podStartE2EDuration="23.128247283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:19.127917397 +0000 UTC m=+23.740501981" watchObservedRunningTime="2026-07-27 19:40:19.128247283 +0000 UTC m=+23.740831866" Jul 27 19:40:19.146957 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.146899 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c845d" podStartSLOduration=4.299223942 podStartE2EDuration="24.146885844s" totalImagesPullingTime="19.847661902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:19.146802804 +0000 UTC m=+23.759387387" watchObservedRunningTime="2026-07-27 19:40:19.146885844 +0000 UTC m=+23.759470427" Jul 27 19:40:19.264198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.264153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:19.365091 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.365012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:19.820017 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.819919 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T19:40:18.858118074Z","UUID":"fe565412-c0df-4285-a5a3-906b2889e663","Handler":null,"Name":"","Endpoint":""} Jul 27 19:40:19.821574 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.821551 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 19:40:19.821698 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.821597 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 19:40:19.869328 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:19.869277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:19.869513 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:19.869444 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:20.870085 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:20.869995 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:20.870551 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:20.870119 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:20.870551 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:20.870149 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:20.870551 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:20.870220 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:21.122810 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:21.122721 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" event={"ID":"a7782c74-246e-4a12-9312-b84a6e64667a","Type":"ContainerStarted","Data":"ba9dac46ded0ee0293a05c7fc47a64e1af78d4034b031acafd2b8a983a0289ec"} Jul 27 19:40:21.125036 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:21.125003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" event={"ID":"52ef6606-39e7-4c70-8386-086a93bf5393","Type":"ContainerStarted","Data":"8ae8303bad94a56356f9d6b772d8ac5f394d89eb1defeea7607ff0a8ec5af82c"} Jul 27 19:40:21.143859 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:21.143814 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcf86" podStartSLOduration=3.458640245 podStartE2EDuration="26.143800439s" totalImagesPullingTime="22.685160194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:21.143402902 +0000 UTC m=+25.755987509" watchObservedRunningTime="2026-07-27 19:40:21.143800439 +0000 UTC m=+25.756385021" Jul 27 19:40:21.488579 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:21.488546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:21.680100 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:21.680065 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:40:21.680717 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:21.680696 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:40:21.870079 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:21.870006 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:21.870227 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:21.870204 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:22.870418 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:22.869548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:22.870418 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:22.869612 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:22.870418 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:22.869828 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:22.870418 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:22.869894 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:23.131014 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:23.130986 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerStarted","Data":"2b708f3d06b5a980d53f9108b7321dac01c90e85954084ca5c99e19e97dacba3"} Jul 27 19:40:23.140406 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:23.140380 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" event={"ID":"a7782c74-246e-4a12-9312-b84a6e64667a","Type":"ContainerStarted","Data":"9b33bdc6232e0f04e6b543ff4c761c43d5a01fb220b7c5624642cca5fa7dbbe2"} Jul 27 19:40:23.140644 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:23.140627 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:40:23.140701 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:23.140682 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:40:23.140742 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:23.140711 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:40:23.158221 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:23.158194 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:40:23.158424 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:23.158365 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:40:23.182783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:23.182731 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" podStartSLOduration=7.860403937 podStartE2EDuration="28.182716951s" totalImagesPullingTime="20.322313014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:23.182396637 +0000 UTC m=+27.794981241" watchObservedRunningTime="2026-07-27 19:40:23.182716951 +0000 UTC m=+27.795301533" Jul 27 19:40:23.870209 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:23.870132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:23.870400 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:23.870230 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:24.144073 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:24.143985 2573 generic.go:356] "Generic (PLEG): container finished" podID="1873aafc-ad68-4fc5-9c04-f6070959451b" containerID="2b708f3d06b5a980d53f9108b7321dac01c90e85954084ca5c99e19e97dacba3" exitCode=0 Jul 27 19:40:24.144855 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:24.144069 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerDied","Data":"2b708f3d06b5a980d53f9108b7321dac01c90e85954084ca5c99e19e97dacba3"} Jul 27 19:40:24.870252 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:24.870064 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:24.870432 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:24.870120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:24.870432 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:24.870383 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:24.870515 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:24.870489 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:25.036136 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.036055 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-69k4m"] Jul 27 19:40:25.040474 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.039065 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q2r5h"] Jul 27 19:40:25.040474 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.039951 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-md4vm"] Jul 27 19:40:25.040474 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.040060 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:25.040474 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:25.040157 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:25.148653 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.148618 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerStarted","Data":"0ac961cad41453ba30d3fd343d22bcf9ef9e5df52ac7a7895d059dd32ff3de39"} Jul 27 19:40:25.148653 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.148653 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:25.149029 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.148664 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:25.149029 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:25.148764 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:25.149261 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:25.149235 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:25.380354 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.380277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:25.433696 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.433668 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:25.493988 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.493968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:25.550083 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.550062 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:40:25.550187 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.550175 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 19:40:25.550739 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.550720 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fs6t5" Jul 27 19:40:25.617696 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.617673 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:25.787442 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.787407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:25.887155 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:25.887123 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:26.153502 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.153427 2573 generic.go:356] "Generic (PLEG): container finished" podID="1873aafc-ad68-4fc5-9c04-f6070959451b" containerID="0ac961cad41453ba30d3fd343d22bcf9ef9e5df52ac7a7895d059dd32ff3de39" exitCode=0 Jul 27 19:40:26.153937 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.153509 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerDied","Data":"0ac961cad41453ba30d3fd343d22bcf9ef9e5df52ac7a7895d059dd32ff3de39"} Jul 27 19:40:26.309278 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.309247 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:26.401633 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.401593 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:26.497659 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.497628 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:26.602851 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.602799 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:26.699328 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.699243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:26.761446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.761335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:26.859085 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.859051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:26.869737 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.869711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:26.869895 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.869805 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:26.869961 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.869906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:26.869961 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:26.869931 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:26.870065 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:26.869989 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:26.870065 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:26.870041 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:26.981092 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:26.981068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:27.087585 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:27.087520 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:28.160344 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.160313 2573 generic.go:356] "Generic (PLEG): container finished" podID="1873aafc-ad68-4fc5-9c04-f6070959451b" containerID="d5daf5ca465d9dceca5e570ed6bf3542d463aa8d7207c25ac7c0f6fdfdbf8bb6" exitCode=0 Jul 27 19:40:28.160702 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.160366 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerDied","Data":"d5daf5ca465d9dceca5e570ed6bf3542d463aa8d7207c25ac7c0f6fdfdbf8bb6"} Jul 27 19:40:28.598657 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.598620 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:28.598657 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.598659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:28.598876 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.598695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:28.598876 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.598788 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:40:28.598876 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.598789 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:40:28.598876 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.598810 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:40:28.598876 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.598835 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:40:28.598876 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.598842 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret podName:f9911a42-c569-4d6b-89e9-08a1551fd7cc nodeName:}" failed. No retries permitted until 2026-07-27 19:41:00.598823712 +0000 UTC m=+65.211408273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret") pod "global-pull-secret-syncer-69k4m" (UID: "f9911a42-c569-4d6b-89e9-08a1551fd7cc") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:40:28.598876 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.598848 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2nhpw for pod openshift-network-diagnostics/network-check-target-md4vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:40:28.598876 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.598858 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs podName:d461523b-81bb-4af6-a2cf-1c77074efbb6 nodeName:}" failed. No retries permitted until 2026-07-27 19:41:00.598851185 +0000 UTC m=+65.211435746 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs") pod "network-metrics-daemon-q2r5h" (UID: "d461523b-81bb-4af6-a2cf-1c77074efbb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:40:28.599226 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.598897 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw podName:f736b83c-af3d-47fb-a7d2-e3349cd84e30 nodeName:}" failed. No retries permitted until 2026-07-27 19:41:00.598881181 +0000 UTC m=+65.211465745 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2nhpw" (UniqueName: "kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw") pod "network-check-target-md4vm" (UID: "f736b83c-af3d-47fb-a7d2-e3349cd84e30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:40:28.644339 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.644308 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:28.747609 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.747571 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:28.771542 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.771515 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-55zm9_087c69da-4dc1-4b01-9b0b-f972323a4124/dns-node-resolver/0.log" Jul 27 19:40:28.869176 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.869095 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:28.869361 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.869209 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:28.869361 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.869211 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:28.869361 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:28.869214 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:28.869361 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.869314 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:28.869575 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:28.869417 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:29.525400 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:29.525142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:30.171214 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:30.171185 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wmbgb_5cf1c32f-1dc1-45e1-8992-6443f5ea0df1/node-ca/0.log" Jul 27 19:40:30.869567 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:30.869538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:30.869975 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:30.869538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:30.869975 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:30.869538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:30.869975 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:30.869730 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:30.869975 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:30.869672 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:30.869975 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:30.869793 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:32.171340 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:32.171315 2573 generic.go:356] "Generic (PLEG): container finished" podID="1873aafc-ad68-4fc5-9c04-f6070959451b" containerID="632f468843e2482ca52f3ce23d9aa839340449e6ed868c1f44e12e254b6eeed6" exitCode=0 Jul 27 19:40:32.171630 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:32.171367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerDied","Data":"632f468843e2482ca52f3ce23d9aa839340449e6ed868c1f44e12e254b6eeed6"} Jul 27 19:40:32.869456 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:32.869424 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:32.869636 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:32.869436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:32.869636 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:32.869549 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:32.869636 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:32.869599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:32.869791 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:32.869667 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:32.869839 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:32.869779 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:33.176273 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:33.176196 2573 generic.go:356] "Generic (PLEG): container finished" podID="1873aafc-ad68-4fc5-9c04-f6070959451b" containerID="e616dfbf24a8a6d0c911b308430a617b48898a715f42152ec796ff40eeefde44" exitCode=0 Jul 27 19:40:33.176273 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:33.176253 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerDied","Data":"e616dfbf24a8a6d0c911b308430a617b48898a715f42152ec796ff40eeefde44"} Jul 27 19:40:33.332985 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:33.332951 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:33.585702 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:33.585665 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:33.727291 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:33.727260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:33.866740 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:33.866643 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:33.967634 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:33.967588 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:34.067799 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:34.067739 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:34.181866 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:34.181837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" event={"ID":"1873aafc-ad68-4fc5-9c04-f6070959451b","Type":"ContainerStarted","Data":"5c685239b169701436edb6d7ac1d8cc7eb96399c0c429865154da1a0399ac06a"} Jul 27 19:40:34.209134 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:34.209081 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wm7nj" podStartSLOduration=8.996144254 podStartE2EDuration="39.209065335s" totalImagesPullingTime="30.212921081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:34.208684137 +0000 UTC m=+38.821268720" watchObservedRunningTime="2026-07-27 19:40:34.209065335 +0000 UTC m=+38.821649917" Jul 27 19:40:34.788913 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:34.788882 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:34.869437 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:34.869398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:34.869437 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:34.869426 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:34.869657 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:34.869398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:34.869657 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:34.869523 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:34.869657 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:34.869568 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:34.869657 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:34.869636 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:34.889960 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:34.889933 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:35.033027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:35.032998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:35.426763 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:35.426729 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:35.631610 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:35.631568 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:35.830974 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:35.830907 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:36.228259 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:36.228222 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:36.835460 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:36.835430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:36.869745 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:36.869726 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:36.869863 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:36.869729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:36.869863 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:36.869831 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:36.869975 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:36.869897 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:36.869975 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:36.869729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:36.869975 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:36.869970 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:37.227611 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:37.227572 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:37.327859 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:37.327817 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:37.558684 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:37.558584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:37.660392 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:37.660361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:37.834764 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:37.834634 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:38.250835 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:38.250783 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:38.365462 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:38.365412 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:38.837158 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:38.837105 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:38.869103 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:38.869074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:38.869243 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:38.869074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:38.869243 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:38.869188 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:38.869243 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:38.869228 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:38.869370 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:38.869074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:38.869370 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:38.869321 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:39.438826 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:39.438792 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:39.539888 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:39.539852 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:39.677132 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:39.677091 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:39.812459 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:39.812376 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:39.913726 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:39.913692 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:40.224072 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:40.224032 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:40.869439 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:40.869402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:40.869439 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:40.869419 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:40.869879 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:40.869503 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:40.869879 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:40.869533 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:40.869879 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:40.869583 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:40.869879 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:40.869670 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:41.031553 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:41.031517 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:41.302954 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:41.302924 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:41.431580 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:41.431546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:41.602787 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:41.602689 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:41.703385 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:41.703354 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:41.772894 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:41.772864 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:41.846446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:41.846416 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:41.913676 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:41.913608 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:42.032843 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:42.032813 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:42.265632 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:42.265603 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:42.330867 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:42.330809 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:42.666240 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:42.666152 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:42.721241 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:42.721203 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:42.869496 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:42.869466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:42.869637 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:42.869466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:42.869637 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:42.869561 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:42.869637 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:42.869466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:42.869779 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:42.869640 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:42.869779 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:42.869702 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:43.228768 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:43.228729 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:43.328760 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:43.328724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:43.429412 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:43.429380 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:43.529884 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:43.529810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:43.647396 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:43.647361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:43.831040 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:43.830956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:44.030349 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:44.030321 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:44.303098 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:44.303070 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:44.403368 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:44.403283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:44.458637 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:44.458603 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:44.869517 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:44.869488 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:44.869678 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:44.869575 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:44.869678 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:44.869577 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:44.869678 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:44.869605 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:44.869678 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:44.869640 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:44.869810 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:44.869721 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:46.869884 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:46.869851 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:46.869884 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:46.869880 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:46.870286 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:46.869852 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:46.870286 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:46.869944 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:46.870286 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:46.870026 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:46.870286 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:46.870084 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:47.404914 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:47.404878 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:47.455680 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:47.455654 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:48.870002 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:48.869971 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:48.870408 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:48.870060 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:48.870408 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:48.870068 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:48.870408 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:48.870087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:48.870408 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:48.870177 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:48.870408 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:48.870246 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:49.037670 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:49.037633 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:50.869999 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:50.869963 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:50.869999 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:50.869979 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:50.870600 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:50.869972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:50.870600 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:50.870062 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:50.870600 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:50.870169 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:50.870600 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:50.870234 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:52.815497 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:52.815471 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:52.869722 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:52.869690 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:52.869840 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:52.869780 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:52.869888 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:52.869876 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2r5h" podUID="d461523b-81bb-4af6-a2cf-1c77074efbb6" Jul 27 19:40:52.869941 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:52.869783 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69k4m" podUID="f9911a42-c569-4d6b-89e9-08a1551fd7cc" Jul 27 19:40:52.869941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:52.869786 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:52.870006 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:52.869962 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-md4vm" podUID="f736b83c-af3d-47fb-a7d2-e3349cd84e30" Jul 27 19:40:52.911675 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:52.911645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:53.065899 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.065818 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:53.187789 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.187764 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-6.ec2.internal" event="NodeReady" Jul 27 19:40:53.187964 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.187900 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 19:40:53.189238 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.189216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:53.264376 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.264332 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5gvl7"] Jul 27 19:40:53.266198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.266180 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tzhgg"] Jul 27 19:40:53.266338 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.266323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5gvl7" Jul 27 19:40:53.267999 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.267978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.268956 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.268939 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 19:40:53.269242 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.269228 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cqn6v\"" Jul 27 19:40:53.269982 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.269969 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 19:40:53.270974 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.270960 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 19:40:53.271224 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.271211 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 19:40:53.271478 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.271463 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 19:40:53.271538 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.271495 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-csj86\"" Jul 27 19:40:53.271915 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.271894 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 19:40:53.272403 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.272390 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 19:40:53.285776 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.285751 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5gvl7"] Jul 27 19:40:53.286510 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.286493 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tzhgg"] Jul 27 19:40:53.355694 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.355615 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-crfhl"] Jul 27 19:40:53.357525 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.357508 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.360618 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.360594 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 19:40:53.360743 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.360639 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 19:40:53.360743 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.360594 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tnv29\"" Jul 27 19:40:53.371172 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.371050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqf7\" (UniqueName: \"kubernetes.io/projected/c17ba55d-0199-45e1-b144-837dd9760c6e-kube-api-access-msqf7\") pod \"ingress-canary-5gvl7\" (UID: \"c17ba55d-0199-45e1-b144-837dd9760c6e\") " pod="openshift-ingress-canary/ingress-canary-5gvl7" Jul 27 19:40:53.371172 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.371088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/baa95d10-67aa-49d1-8e70-a7d0e2652e95-crio-socket\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.371172 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.371165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baa95d10-67aa-49d1-8e70-a7d0e2652e95-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.371413 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.371227 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2h2c\" (UniqueName: \"kubernetes.io/projected/baa95d10-67aa-49d1-8e70-a7d0e2652e95-kube-api-access-s2h2c\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.371413 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.371274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c17ba55d-0199-45e1-b144-837dd9760c6e-cert\") pod \"ingress-canary-5gvl7\" (UID: \"c17ba55d-0199-45e1-b144-837dd9760c6e\") " pod="openshift-ingress-canary/ingress-canary-5gvl7" Jul 27 19:40:53.371413 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.371352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baa95d10-67aa-49d1-8e70-a7d0e2652e95-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.371413 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.371380 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/baa95d10-67aa-49d1-8e70-a7d0e2652e95-data-volume\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.371855 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.371828 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-crfhl"] Jul 27 19:40:53.458238 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.458209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:53.472291 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472268 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lbh\" (UniqueName: \"kubernetes.io/projected/093af648-a209-43a7-bfb4-16b0155ae839-kube-api-access-v7lbh\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.472412 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093af648-a209-43a7-bfb4-16b0155ae839-tmp-dir\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.472412 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baa95d10-67aa-49d1-8e70-a7d0e2652e95-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.472412 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2h2c\" (UniqueName: \"kubernetes.io/projected/baa95d10-67aa-49d1-8e70-a7d0e2652e95-kube-api-access-s2h2c\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.472572 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c17ba55d-0199-45e1-b144-837dd9760c6e-cert\") pod \"ingress-canary-5gvl7\" (UID: \"c17ba55d-0199-45e1-b144-837dd9760c6e\") " pod="openshift-ingress-canary/ingress-canary-5gvl7" Jul 27 19:40:53.472572 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/093af648-a209-43a7-bfb4-16b0155ae839-metrics-tls\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.472572 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472521 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baa95d10-67aa-49d1-8e70-a7d0e2652e95-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.472572 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/baa95d10-67aa-49d1-8e70-a7d0e2652e95-data-volume\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.472758 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/093af648-a209-43a7-bfb4-16b0155ae839-config-volume\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.472758 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqf7\" (UniqueName: \"kubernetes.io/projected/c17ba55d-0199-45e1-b144-837dd9760c6e-kube-api-access-msqf7\") pod \"ingress-canary-5gvl7\" (UID: \"c17ba55d-0199-45e1-b144-837dd9760c6e\") " pod="openshift-ingress-canary/ingress-canary-5gvl7" Jul 27 19:40:53.472758 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/baa95d10-67aa-49d1-8e70-a7d0e2652e95-crio-socket\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.472895 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472787 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/baa95d10-67aa-49d1-8e70-a7d0e2652e95-crio-socket\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.472895 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.472849 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/baa95d10-67aa-49d1-8e70-a7d0e2652e95-data-volume\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.473037 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.473016 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/baa95d10-67aa-49d1-8e70-a7d0e2652e95-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.476401 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.476383 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/baa95d10-67aa-49d1-8e70-a7d0e2652e95-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.476546 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.476527 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c17ba55d-0199-45e1-b144-837dd9760c6e-cert\") pod \"ingress-canary-5gvl7\" (UID: \"c17ba55d-0199-45e1-b144-837dd9760c6e\") " pod="openshift-ingress-canary/ingress-canary-5gvl7" Jul 27 19:40:53.482542 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.482523 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqf7\" (UniqueName: \"kubernetes.io/projected/c17ba55d-0199-45e1-b144-837dd9760c6e-kube-api-access-msqf7\") pod \"ingress-canary-5gvl7\" (UID: \"c17ba55d-0199-45e1-b144-837dd9760c6e\") " pod="openshift-ingress-canary/ingress-canary-5gvl7" Jul 27 19:40:53.482644 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.482616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2h2c\" (UniqueName: \"kubernetes.io/projected/baa95d10-67aa-49d1-8e70-a7d0e2652e95-kube-api-access-s2h2c\") pod \"insights-runtime-extractor-tzhgg\" (UID: \"baa95d10-67aa-49d1-8e70-a7d0e2652e95\") " pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.551261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.551236 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:53.572948 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.572924 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lbh\" (UniqueName: \"kubernetes.io/projected/093af648-a209-43a7-bfb4-16b0155ae839-kube-api-access-v7lbh\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.573041 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.572961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093af648-a209-43a7-bfb4-16b0155ae839-tmp-dir\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.573099 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.573042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/093af648-a209-43a7-bfb4-16b0155ae839-metrics-tls\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.573199 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.573180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/093af648-a209-43a7-bfb4-16b0155ae839-config-volume\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.573407 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.573390 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093af648-a209-43a7-bfb4-16b0155ae839-tmp-dir\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.573666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.573645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/093af648-a209-43a7-bfb4-16b0155ae839-config-volume\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.575474 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.575448 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/093af648-a209-43a7-bfb4-16b0155ae839-metrics-tls\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.576249 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.576230 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5gvl7" Jul 27 19:40:53.580903 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.580887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tzhgg" Jul 27 19:40:53.588020 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.587958 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lbh\" (UniqueName: \"kubernetes.io/projected/093af648-a209-43a7-bfb4-16b0155ae839-kube-api-access-v7lbh\") pod \"dns-default-crfhl\" (UID: \"093af648-a209-43a7-bfb4-16b0155ae839\") " pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.626449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.625630 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw"] Jul 27 19:40:53.628286 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.628266 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" Jul 27 19:40:53.631109 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.631087 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 27 19:40:53.631355 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.631338 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vbpv7\"" Jul 27 19:40:53.645918 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.645220 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw"] Jul 27 19:40:53.650637 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.650509 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:53.666552 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.666372 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-crfhl" Jul 27 19:40:53.674089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.674015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/398a7dd4-7a0a-4c74-9402-dd6c6c7857ac-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cz6gw\" (UID: \"398a7dd4-7a0a-4c74-9402-dd6c6c7857ac\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" Jul 27 19:40:53.735116 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.735084 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tzhgg"] Jul 27 19:40:53.741617 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:40:53.741587 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa95d10_67aa_49d1_8e70_a7d0e2652e95.slice/crio-50f7ca4dd0267fb9970e5e7d67f92b55f686fe07faab2216ed3d79ad0fab354f WatchSource:0}: Error finding container 50f7ca4dd0267fb9970e5e7d67f92b55f686fe07faab2216ed3d79ad0fab354f: Status 404 returned error can't find the container with id 50f7ca4dd0267fb9970e5e7d67f92b55f686fe07faab2216ed3d79ad0fab354f Jul 27 19:40:53.750861 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.750835 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5gvl7"] Jul 27 19:40:53.754155 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:40:53.754130 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17ba55d_0199_45e1_b144_837dd9760c6e.slice/crio-37b6fbbff65584ba18f0b18caf8ef1999040cad7bc47c239195af540444910f0 WatchSource:0}: Error finding container 37b6fbbff65584ba18f0b18caf8ef1999040cad7bc47c239195af540444910f0: Status 404 returned error can't find the container with id 37b6fbbff65584ba18f0b18caf8ef1999040cad7bc47c239195af540444910f0 Jul 27 19:40:53.774844 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.774820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/398a7dd4-7a0a-4c74-9402-dd6c6c7857ac-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cz6gw\" (UID: \"398a7dd4-7a0a-4c74-9402-dd6c6c7857ac\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" Jul 27 19:40:53.774974 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:53.774959 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 27 19:40:53.775027 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:40:53.775017 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/398a7dd4-7a0a-4c74-9402-dd6c6c7857ac-tls-certificates podName:398a7dd4-7a0a-4c74-9402-dd6c6c7857ac nodeName:}" failed. No retries permitted until 2026-07-27 19:40:54.275001542 +0000 UTC m=+58.887586103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/398a7dd4-7a0a-4c74-9402-dd6c6c7857ac-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" (UID: "398a7dd4-7a0a-4c74-9402-dd6c6c7857ac") : secret "prometheus-operator-admission-webhook-tls" not found Jul 27 19:40:53.795678 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.795655 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-crfhl"] Jul 27 19:40:53.798939 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:40:53.798918 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093af648_a209_43a7_bfb4_16b0155ae839.slice/crio-8824a90bf554ce854d3b4e9fe337ea221d3ca238555a69d30f7d4613a92dc705 WatchSource:0}: Error finding container 8824a90bf554ce854d3b4e9fe337ea221d3ca238555a69d30f7d4613a92dc705: Status 404 returned error can't find the container with id 8824a90bf554ce854d3b4e9fe337ea221d3ca238555a69d30f7d4613a92dc705 Jul 27 19:40:53.906848 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:53.906764 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:54.229726 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.229687 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crfhl" event={"ID":"093af648-a209-43a7-bfb4-16b0155ae839","Type":"ContainerStarted","Data":"8824a90bf554ce854d3b4e9fe337ea221d3ca238555a69d30f7d4613a92dc705"} Jul 27 19:40:54.231029 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.230998 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzhgg" event={"ID":"baa95d10-67aa-49d1-8e70-a7d0e2652e95","Type":"ContainerStarted","Data":"50f7ca4dd0267fb9970e5e7d67f92b55f686fe07faab2216ed3d79ad0fab354f"} Jul 27 19:40:54.232940 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.232906 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5gvl7" event={"ID":"c17ba55d-0199-45e1-b144-837dd9760c6e","Type":"ContainerStarted","Data":"37b6fbbff65584ba18f0b18caf8ef1999040cad7bc47c239195af540444910f0"} Jul 27 19:40:54.282180 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.280070 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/398a7dd4-7a0a-4c74-9402-dd6c6c7857ac-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cz6gw\" (UID: \"398a7dd4-7a0a-4c74-9402-dd6c6c7857ac\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" Jul 27 19:40:54.286585 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.286553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/398a7dd4-7a0a-4c74-9402-dd6c6c7857ac-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cz6gw\" (UID: \"398a7dd4-7a0a-4c74-9402-dd6c6c7857ac\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" Jul 27 19:40:54.478259 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.477471 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:54.538732 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.538647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" Jul 27 19:40:54.543992 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.543946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:54.745876 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.745839 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw"] Jul 27 19:40:54.869186 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.869099 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:40:54.869186 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.869149 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:40:54.869420 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.869334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:40:54.874337 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.874313 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4fnrz\"" Jul 27 19:40:54.874645 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.874626 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 19:40:54.874839 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.874823 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 19:40:54.875022 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.875005 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 19:40:54.875198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.875183 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 19:40:54.875394 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:54.875379 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-klph9\"" Jul 27 19:40:55.166378 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:55.165379 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fg5b" Jul 27 19:40:55.238383 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:55.238347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" event={"ID":"398a7dd4-7a0a-4c74-9402-dd6c6c7857ac","Type":"ContainerStarted","Data":"dbcd51c7d0da8b8638841eb641a49fc923490cccf3c7b87294528057db3e944b"} Jul 27 19:40:55.284846 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:55.284811 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:55.384114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:55.384083 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:55.441726 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:55.441698 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:56.510680 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:56.510651 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:56.567923 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:56.567886 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:56.911654 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:56.911576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:57.011618 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.011579 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:57.741480 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.741451 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66fc9b4c4b-5vvhz"] Jul 27 19:40:57.751826 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.751798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.756921 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.756899 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 19:40:57.757408 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.757385 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 19:40:57.757718 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.757694 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sj6nc\"" Jul 27 19:40:57.757955 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.757935 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 19:40:57.758075 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.757999 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 19:40:57.758471 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.758449 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 19:40:57.758471 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.758467 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 19:40:57.758652 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.758461 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 19:40:57.761321 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.760190 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fc9b4c4b-5vvhz"] Jul 27 19:40:57.809499 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.809462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-service-ca\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.809663 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.809555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-oauth-serving-cert\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.809663 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.809635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-serving-cert\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.809771 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.809693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsw5\" (UniqueName: \"kubernetes.io/projected/375dc3d2-f175-41b4-81c1-29e98795cd7a-kube-api-access-tnsw5\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.809771 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.809757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-oauth-config\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.809848 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.809804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-config\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.910235 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.910199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-oauth-config\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.910411 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.910260 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-config\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.910411 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.910325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-service-ca\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.910411 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.910350 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-oauth-serving-cert\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.910411 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.910385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-serving-cert\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.910411 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.910408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsw5\" (UniqueName: \"kubernetes.io/projected/375dc3d2-f175-41b4-81c1-29e98795cd7a-kube-api-access-tnsw5\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.911160 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.911123 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-oauth-serving-cert\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.911160 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.911132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-service-ca\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.911351 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.911181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-config\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.913408 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.913388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-serving-cert\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.913621 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.913604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-oauth-config\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:57.919233 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:57.919214 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsw5\" (UniqueName: \"kubernetes.io/projected/375dc3d2-f175-41b4-81c1-29e98795cd7a-kube-api-access-tnsw5\") pod \"console-66fc9b4c4b-5vvhz\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:58.062834 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:58.062617 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:40:58.561268 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:58.561228 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:40:58.662925 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:40:58.662874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:00.110711 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.110659 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:00.201813 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.200469 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fc9b4c4b-5vvhz"] Jul 27 19:41:00.204478 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.204389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:00.209152 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:41:00.209117 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375dc3d2_f175_41b4_81c1_29e98795cd7a.slice/crio-2bbfa819b949f4551322cc76717e870cef35037ff6cdb54cd95da698e7091822 WatchSource:0}: Error finding container 2bbfa819b949f4551322cc76717e870cef35037ff6cdb54cd95da698e7091822: Status 404 returned error can't find the container with id 2bbfa819b949f4551322cc76717e870cef35037ff6cdb54cd95da698e7091822 Jul 27 19:41:00.255054 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.255027 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5gvl7" event={"ID":"c17ba55d-0199-45e1-b144-837dd9760c6e","Type":"ContainerStarted","Data":"525c46a37634168929dfae3b85dd3c8a89d7f5bd4296daf57e202f76da8f0c3a"} Jul 27 19:41:00.256932 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.256902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crfhl" event={"ID":"093af648-a209-43a7-bfb4-16b0155ae839","Type":"ContainerStarted","Data":"3b98c736ec891fd129036d624a294b6d986a9c5fb6e2a6d4cefcd4c92e71eb20"} Jul 27 19:41:00.259562 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.259536 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzhgg" event={"ID":"baa95d10-67aa-49d1-8e70-a7d0e2652e95","Type":"ContainerStarted","Data":"4e8c2c3e279fed797eafc0d7aa3e1ec0f5539f85046697d02f2253274ae51018"} Jul 27 19:41:00.261286 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.261260 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" event={"ID":"398a7dd4-7a0a-4c74-9402-dd6c6c7857ac","Type":"ContainerStarted","Data":"fe07d38631420109f83e330beb6506e6866d9bb6156409b7465ca22beb0114fe"} Jul 27 19:41:00.261420 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.261404 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" Jul 27 19:41:00.262773 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.262753 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fc9b4c4b-5vvhz" event={"ID":"375dc3d2-f175-41b4-81c1-29e98795cd7a","Type":"ContainerStarted","Data":"2bbfa819b949f4551322cc76717e870cef35037ff6cdb54cd95da698e7091822"} Jul 27 19:41:00.266982 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.266963 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" Jul 27 19:41:00.293016 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.292903 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5gvl7" podStartSLOduration=1.011285129 podStartE2EDuration="7.292880802s" totalImagesPullingTime="6.281595673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:00.272316607 +0000 UTC m=+64.884901185" watchObservedRunningTime="2026-07-27 19:41:00.292880802 +0000 UTC m=+64.905465388" Jul 27 19:41:00.293159 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.293035 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cz6gw" podStartSLOduration=2.001734449 podStartE2EDuration="7.293026427s" totalImagesPullingTime="5.291291978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:00.291583783 +0000 UTC m=+64.904168369" watchObservedRunningTime="2026-07-27 19:41:00.293026427 +0000 UTC m=+64.905611008" Jul 27 19:41:00.634954 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.634872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:41:00.634954 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.634927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:41:00.634954 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.634948 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:41:00.637525 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.637502 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 19:41:00.637611 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.637539 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 19:41:00.637611 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.637602 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 19:41:00.647551 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.647527 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 19:41:00.647657 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.647590 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9911a42-c569-4d6b-89e9-08a1551fd7cc-original-pull-secret\") pod \"global-pull-secret-syncer-69k4m\" (UID: \"f9911a42-c569-4d6b-89e9-08a1551fd7cc\") " pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:41:00.647742 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.647725 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d461523b-81bb-4af6-a2cf-1c77074efbb6-metrics-certs\") pod \"network-metrics-daemon-q2r5h\" (UID: \"d461523b-81bb-4af6-a2cf-1c77074efbb6\") " pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:41:00.656205 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.656176 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:00.658141 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.658118 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nhpw\" (UniqueName: \"kubernetes.io/projected/f736b83c-af3d-47fb-a7d2-e3349cd84e30-kube-api-access-2nhpw\") pod \"network-check-target-md4vm\" (UID: \"f736b83c-af3d-47fb-a7d2-e3349cd84e30\") " pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:41:00.702403 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.702360 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-hk4nc"] Jul 27 19:41:00.705661 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.705637 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.709915 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.709888 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 19:41:00.710055 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.709900 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 19:41:00.710175 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.710161 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-94pxl\"" Jul 27 19:41:00.710245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.710179 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 19:41:00.710335 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.710321 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 27 19:41:00.710425 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.710403 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 27 19:41:00.716458 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.716432 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-hk4nc"] Jul 27 19:41:00.743118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.743086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:00.836660 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.836457 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9d57a89-726b-4797-a5b4-b20af5d66859-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.836660 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.836497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xh2\" (UniqueName: \"kubernetes.io/projected/b9d57a89-726b-4797-a5b4-b20af5d66859-kube-api-access-66xh2\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.836660 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.836520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9d57a89-726b-4797-a5b4-b20af5d66859-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.836660 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.836573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9d57a89-726b-4797-a5b4-b20af5d66859-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.890746 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.890493 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-klph9\"" Jul 27 19:41:00.897248 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.897052 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4fnrz\"" Jul 27 19:41:00.898820 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.898542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:41:00.902357 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.902169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69k4m" Jul 27 19:41:00.904870 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.904728 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2r5h" Jul 27 19:41:00.932527 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.932498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:00.937051 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.937027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9d57a89-726b-4797-a5b4-b20af5d66859-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.937187 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.937100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9d57a89-726b-4797-a5b4-b20af5d66859-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.937187 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.937134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9d57a89-726b-4797-a5b4-b20af5d66859-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.937187 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.937164 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xh2\" (UniqueName: \"kubernetes.io/projected/b9d57a89-726b-4797-a5b4-b20af5d66859-kube-api-access-66xh2\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.938243 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.938215 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9d57a89-726b-4797-a5b4-b20af5d66859-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.940805 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.940772 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9d57a89-726b-4797-a5b4-b20af5d66859-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.949351 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.949280 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xh2\" (UniqueName: \"kubernetes.io/projected/b9d57a89-726b-4797-a5b4-b20af5d66859-kube-api-access-66xh2\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:00.950994 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:00.950947 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9d57a89-726b-4797-a5b4-b20af5d66859-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hk4nc\" (UID: \"b9d57a89-726b-4797-a5b4-b20af5d66859\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:01.020545 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.020337 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" Jul 27 19:41:01.038034 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.037909 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:01.124828 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.124755 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-md4vm"] Jul 27 19:41:01.142898 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.142868 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q2r5h"] Jul 27 19:41:01.173448 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.173417 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-69k4m"] Jul 27 19:41:01.248083 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.248049 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-hk4nc"] Jul 27 19:41:01.253068 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:41:01.253029 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d57a89_726b_4797_a5b4_b20af5d66859.slice/crio-bff0bd4ea14ac61adde738e0ff01d3fb59cda23db898b9256837ccd128bb5baf WatchSource:0}: Error finding container bff0bd4ea14ac61adde738e0ff01d3fb59cda23db898b9256837ccd128bb5baf: Status 404 returned error can't find the container with id bff0bd4ea14ac61adde738e0ff01d3fb59cda23db898b9256837ccd128bb5baf Jul 27 19:41:01.277104 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.277065 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-69k4m" event={"ID":"f9911a42-c569-4d6b-89e9-08a1551fd7cc","Type":"ContainerStarted","Data":"6272e9621c8ef093256b88530e8c4ed2ca60988ed28273cbc709af01d86f5f0e"} Jul 27 19:41:01.279827 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.278929 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" event={"ID":"b9d57a89-726b-4797-a5b4-b20af5d66859","Type":"ContainerStarted","Data":"bff0bd4ea14ac61adde738e0ff01d3fb59cda23db898b9256837ccd128bb5baf"} Jul 27 19:41:01.281731 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.281593 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2r5h" event={"ID":"d461523b-81bb-4af6-a2cf-1c77074efbb6","Type":"ContainerStarted","Data":"0ea084ab2b77c21b8a8d66519f4265abb7b3f66ae7705887715a5d74b2bb8133"} Jul 27 19:41:01.285507 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.285445 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-md4vm" event={"ID":"f736b83c-af3d-47fb-a7d2-e3349cd84e30","Type":"ContainerStarted","Data":"a1c311268ff7a1c82e636a341f13939f9326997813cc5c711dab6447a7774ce5"} Jul 27 19:41:01.288287 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.288266 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crfhl" event={"ID":"093af648-a209-43a7-bfb4-16b0155ae839","Type":"ContainerStarted","Data":"5f232e85c3194a7815284a474146a1021b1c99bd9fbe12b93046b17215927644"} Jul 27 19:41:01.289101 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.289082 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-crfhl" Jul 27 19:41:01.310076 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.309659 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-crfhl" podStartSLOduration=2.358132992 podStartE2EDuration="8.309638083s" totalImagesPullingTime="5.951505091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:01.308767516 +0000 UTC m=+65.921352117" watchObservedRunningTime="2026-07-27 19:41:01.309638083 +0000 UTC m=+65.922222686" Jul 27 19:41:01.416918 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:01.416834 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:03.314480 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:03.314451 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:03.417211 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:03.417138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:04.302783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.302746 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c8898f6bb-5f57l"] Jul 27 19:41:04.311573 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.311542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.319011 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.318981 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c8898f6bb-5f57l"] Jul 27 19:41:04.322263 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.321802 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 19:41:04.371138 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.371103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-oauth-config\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.371345 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.371152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-oauth-serving-cert\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.371345 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.371181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-service-ca\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.371345 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.371214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-serving-cert\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.371345 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.371242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-trusted-ca-bundle\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.371345 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.371272 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-config\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.371345 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.371320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27c8\" (UniqueName: \"kubernetes.io/projected/09de69fa-ce00-4850-a64f-3e11ebdb0f16-kube-api-access-r27c8\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.472616 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.472584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-config\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.472616 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.472632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27c8\" (UniqueName: \"kubernetes.io/projected/09de69fa-ce00-4850-a64f-3e11ebdb0f16-kube-api-access-r27c8\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.472851 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.472776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-oauth-config\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.472851 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.472827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-oauth-serving-cert\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.472964 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.472863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-service-ca\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.472964 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.472893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-serving-cert\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.472964 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.472924 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-trusted-ca-bundle\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.474042 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.473435 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-config\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.474042 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.473630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-oauth-serving-cert\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.474042 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.473863 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-trusted-ca-bundle\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.474042 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.474002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-service-ca\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.476276 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.476251 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-serving-cert\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.476662 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.476637 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-oauth-config\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.481898 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.481786 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27c8\" (UniqueName: \"kubernetes.io/projected/09de69fa-ce00-4850-a64f-3e11ebdb0f16-kube-api-access-r27c8\") pod \"console-6c8898f6bb-5f57l\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:04.625350 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:04.625236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:05.590838 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:05.590800 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:05.691060 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:05.691016 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:08.135394 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:08.135357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:08.197643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:08.197605 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:09.146164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.146095 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c8898f6bb-5f57l"] Jul 27 19:41:09.159749 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.159687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:09.259378 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.259345 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:09.319180 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.318908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzhgg" event={"ID":"baa95d10-67aa-49d1-8e70-a7d0e2652e95","Type":"ContainerStarted","Data":"682a093c283111b5b9240b229591009ee1428235d0de12e8109ac0067e84c789"} Jul 27 19:41:09.322441 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.322392 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fc9b4c4b-5vvhz" event={"ID":"375dc3d2-f175-41b4-81c1-29e98795cd7a","Type":"ContainerStarted","Data":"7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367"} Jul 27 19:41:09.324607 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.324113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-69k4m" event={"ID":"f9911a42-c569-4d6b-89e9-08a1551fd7cc","Type":"ContainerStarted","Data":"5895dd05208d729a9f63c323509d6a0f9caf7c19409bb37db558f68e0f0c1708"} Jul 27 19:41:09.326758 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.326729 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" event={"ID":"b9d57a89-726b-4797-a5b4-b20af5d66859","Type":"ContainerStarted","Data":"b6addd765bb5df78175844b45a85b6f5f18b913dc08415aabf88468614a289a1"} Jul 27 19:41:09.326876 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.326811 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" event={"ID":"b9d57a89-726b-4797-a5b4-b20af5d66859","Type":"ContainerStarted","Data":"386ce561e86f88155685fa6935bbfeea26b7d6200237b7cf53e01a624fc06705"} Jul 27 19:41:09.329269 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.329245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2r5h" event={"ID":"d461523b-81bb-4af6-a2cf-1c77074efbb6","Type":"ContainerStarted","Data":"fcf76a6c43618b34c042476a85da15309c7880cc055408ccf941088c4c289740"} Jul 27 19:41:09.329406 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.329322 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2r5h" event={"ID":"d461523b-81bb-4af6-a2cf-1c77074efbb6","Type":"ContainerStarted","Data":"835d130cdc0203cda3e79c3bb59bad6228b7ba4ba24c4396297383407c0abcc5"} Jul 27 19:41:09.331420 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.331363 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8898f6bb-5f57l" event={"ID":"09de69fa-ce00-4850-a64f-3e11ebdb0f16","Type":"ContainerStarted","Data":"61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d"} Jul 27 19:41:09.331420 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.331399 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8898f6bb-5f57l" event={"ID":"09de69fa-ce00-4850-a64f-3e11ebdb0f16","Type":"ContainerStarted","Data":"865dff5664575020dc25510a4d19271b05c63c854b65d0bc03c5d25e77ab561d"} Jul 27 19:41:09.332635 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.332610 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-md4vm" event={"ID":"f736b83c-af3d-47fb-a7d2-e3349cd84e30","Type":"ContainerStarted","Data":"6b874452d75de6e4cbaf970ac58a280ba545a0e0788243756853af073b4a4345"} Jul 27 19:41:09.332760 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.332670 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:41:09.343061 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.343009 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66fc9b4c4b-5vvhz" podStartSLOduration=3.594234783 podStartE2EDuration="12.342989751s" totalImagesPullingTime="8.748754968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:09.342679224 +0000 UTC m=+73.955263831" watchObservedRunningTime="2026-07-27 19:41:09.342989751 +0000 UTC m=+73.955574332" Jul 27 19:41:09.346909 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.346884 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:09.360713 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.360664 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-hk4nc" podStartSLOduration=1.650300037 podStartE2EDuration="9.360645975s" totalImagesPullingTime="7.710345938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:09.359892805 +0000 UTC m=+73.972477388" watchObservedRunningTime="2026-07-27 19:41:09.360645975 +0000 UTC m=+73.973230559" Jul 27 19:41:09.395041 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.394967 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-md4vm" podStartSLOduration=66.56986066 podStartE2EDuration="1m14.394949063s" totalImagesPullingTime="7.825088403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:09.394682665 +0000 UTC m=+74.007267248" watchObservedRunningTime="2026-07-27 19:41:09.394949063 +0000 UTC m=+74.007533644" Jul 27 19:41:09.395220 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.395200 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c8898f6bb-5f57l" podStartSLOduration=5.395192733 podStartE2EDuration="5.395192733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:09.377395801 +0000 UTC m=+73.989980385" watchObservedRunningTime="2026-07-27 19:41:09.395192733 +0000 UTC m=+74.007777318" Jul 27 19:41:09.420622 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.420536 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q2r5h" podStartSLOduration=67.244231259 podStartE2EDuration="1m14.42052157s" totalImagesPullingTime="7.176290311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:09.419881966 +0000 UTC m=+74.032466545" watchObservedRunningTime="2026-07-27 19:41:09.42052157 +0000 UTC m=+74.033106153" Jul 27 19:41:09.426694 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.426668 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:09.437314 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.437200 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-69k4m" podStartSLOduration=65.652359334 podStartE2EDuration="1m13.437186381s" totalImagesPullingTime="7.784827047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:09.436946793 +0000 UTC m=+74.049531386" watchObservedRunningTime="2026-07-27 19:41:09.437186381 +0000 UTC m=+74.049770942" Jul 27 19:41:09.529030 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.528999 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:09.639615 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:09.639563 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:10.837848 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:10.837812 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:11.038964 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.038930 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:11.078727 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.078696 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756"] Jul 27 19:41:11.083825 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.083802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.086340 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.086317 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 27 19:41:11.086453 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.086317 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p6g2h\"" Jul 27 19:41:11.087571 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.087550 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 27 19:41:11.101352 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.101283 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756"] Jul 27 19:41:11.102317 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.102279 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l"] Jul 27 19:41:11.105562 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.105541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.108540 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.108504 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 27 19:41:11.108621 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.108588 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 27 19:41:11.108749 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.108730 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cqhfd\"" Jul 27 19:41:11.108809 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.108784 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 27 19:41:11.138231 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.137764 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l"] Jul 27 19:41:11.139827 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.139806 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rk58b"] Jul 27 19:41:11.143144 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.143127 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.146041 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.146023 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rvkzp\"" Jul 27 19:41:11.146872 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.146857 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 19:41:11.146994 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.146978 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 19:41:11.147096 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.147081 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 19:41:11.161043 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.161014 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:11.232004 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.231969 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.232183 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88007507-fc7d-4645-973d-2eb422dabc19-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.232183 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctlbk\" (UniqueName: \"kubernetes.io/projected/88007507-fc7d-4645-973d-2eb422dabc19-kube-api-access-ctlbk\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.232183 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.232183 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-accelerators-collector-config\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.232183 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvnz\" (UniqueName: \"kubernetes.io/projected/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-api-access-7dvnz\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.232183 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j5wv\" (UniqueName: \"kubernetes.io/projected/37dda389-7aa6-461e-8fcb-9d00ae6dde05-kube-api-access-8j5wv\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.232479 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232206 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-wtmp\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.232479 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.232479 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232310 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88007507-fc7d-4645-973d-2eb422dabc19-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.232479 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37dda389-7aa6-461e-8fcb-9d00ae6dde05-sys\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.232479 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232385 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88007507-fc7d-4645-973d-2eb422dabc19-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.232479 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-textfile\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.232479 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.232769 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232496 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37dda389-7aa6-461e-8fcb-9d00ae6dde05-metrics-client-ca\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.232769 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.232769 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232557 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/37dda389-7aa6-461e-8fcb-9d00ae6dde05-root\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.232769 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-tls\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.232769 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.232625 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.266415 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.266386 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:11.333833 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.333796 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.334050 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.333844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/37dda389-7aa6-461e-8fcb-9d00ae6dde05-root\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.334050 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.333870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-tls\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.334050 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.333900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.334050 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.333929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.334050 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.333950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/37dda389-7aa6-461e-8fcb-9d00ae6dde05-root\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.334050 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.333956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88007507-fc7d-4645-973d-2eb422dabc19-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.334050 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctlbk\" (UniqueName: \"kubernetes.io/projected/88007507-fc7d-4645-973d-2eb422dabc19-kube-api-access-ctlbk\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-accelerators-collector-config\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvnz\" (UniqueName: \"kubernetes.io/projected/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-api-access-7dvnz\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j5wv\" (UniqueName: \"kubernetes.io/projected/37dda389-7aa6-461e-8fcb-9d00ae6dde05-kube-api-access-8j5wv\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-wtmp\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88007507-fc7d-4645-973d-2eb422dabc19-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37dda389-7aa6-461e-8fcb-9d00ae6dde05-sys\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334326 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88007507-fc7d-4645-973d-2eb422dabc19-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-textfile\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.335128 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334540 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-wtmp\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.335128 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334913 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.335128 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.334989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37dda389-7aa6-461e-8fcb-9d00ae6dde05-sys\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.335128 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.335017 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-accelerators-collector-config\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.335128 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.335072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37dda389-7aa6-461e-8fcb-9d00ae6dde05-metrics-client-ca\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.335396 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.335187 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.335396 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.335228 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.336109 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.335625 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37dda389-7aa6-461e-8fcb-9d00ae6dde05-metrics-client-ca\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.336109 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.335974 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-textfile\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.336109 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.335974 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88007507-fc7d-4645-973d-2eb422dabc19-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.337962 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.337784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.337962 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.337887 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.338205 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.338181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/37dda389-7aa6-461e-8fcb-9d00ae6dde05-node-exporter-tls\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.338354 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.338334 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88007507-fc7d-4645-973d-2eb422dabc19-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.338566 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.338521 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88007507-fc7d-4645-973d-2eb422dabc19-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.340170 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.340146 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.346180 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.346104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctlbk\" (UniqueName: \"kubernetes.io/projected/88007507-fc7d-4645-973d-2eb422dabc19-kube-api-access-ctlbk\") pod \"openshift-state-metrics-b8f49f5c6-tx756\" (UID: \"88007507-fc7d-4645-973d-2eb422dabc19\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.346767 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.346743 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j5wv\" (UniqueName: \"kubernetes.io/projected/37dda389-7aa6-461e-8fcb-9d00ae6dde05-kube-api-access-8j5wv\") pod \"node-exporter-rk58b\" (UID: \"37dda389-7aa6-461e-8fcb-9d00ae6dde05\") " pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.346920 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.346873 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvnz\" (UniqueName: \"kubernetes.io/projected/7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8-kube-api-access-7dvnz\") pod \"kube-state-metrics-56d9f944b4-zr86l\" (UID: \"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.367543 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.367509 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:11.394815 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.394589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" Jul 27 19:41:11.417387 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.417364 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" Jul 27 19:41:11.452471 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.452441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rk58b" Jul 27 19:41:11.486864 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.486826 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:11.726920 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:11.726884 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:12.027742 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.027663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:12.128898 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.128855 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:12.173708 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.173673 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:41:12.189221 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.188008 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.191227 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.191191 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 27 19:41:12.191513 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.191271 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 27 19:41:12.192729 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.192030 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 27 19:41:12.192729 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.192170 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 27 19:41:12.192729 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.192239 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 27 19:41:12.192729 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.192364 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 27 19:41:12.192729 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.192441 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jws6n\"" Jul 27 19:41:12.192729 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.192542 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 27 19:41:12.192729 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.192567 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:41:12.192729 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.192603 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 27 19:41:12.193105 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.192751 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 27 19:41:12.195519 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.195494 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:12.299404 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.299327 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-crfhl" Jul 27 19:41:12.324882 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.324848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:12.344428 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344623 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptb4\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-kube-api-access-kptb4\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344623 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344474 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344623 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-config-out\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344623 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344623 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344623 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344675 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.344941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.344884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-web-config\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.445880 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.445829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446062 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.445911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446062 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.445958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-web-config\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446062 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.445985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446062 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.446012 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptb4\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-kube-api-access-kptb4\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446062 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.446032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446062 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.446055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-config-out\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446379 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.446081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446379 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.446108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446379 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.446134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446379 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.446166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446379 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.446202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.446379 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.446227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.447251 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.447222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.448871 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.448235 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.449428 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.449400 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.452621 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.452597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-config-out\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.452783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.452761 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.453021 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.453002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.453064 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.453012 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.453064 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.453053 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.454150 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.453462 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.454150 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.453746 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-web-config\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.454150 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.453953 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.455004 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.454652 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.468502 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.468473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptb4\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-kube-api-access-kptb4\") pod \"alertmanager-main-0\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.500929 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.500897 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:41:12.545076 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.545044 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:12.602455 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.602415 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:12.603654 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:41:12.603620 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37dda389_7aa6_461e_8fcb_9d00ae6dde05.slice/crio-28593a9c9f469f134b384a52b7855126d3ff7cab07c6ed704e32ad848ae09188 WatchSource:0}: Error finding container 28593a9c9f469f134b384a52b7855126d3ff7cab07c6ed704e32ad848ae09188: Status 404 returned error can't find the container with id 28593a9c9f469f134b384a52b7855126d3ff7cab07c6ed704e32ad848ae09188 Jul 27 19:41:12.758998 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.758972 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l"] Jul 27 19:41:12.761348 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:41:12.761327 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9d2e21_5dff_4ad8_bf15_e3bd4f44e2b8.slice/crio-425a75dabf8a82d77ed9c50f6a18e36d084afa707cbc78c1ec12885611190077 WatchSource:0}: Error finding container 425a75dabf8a82d77ed9c50f6a18e36d084afa707cbc78c1ec12885611190077: Status 404 returned error can't find the container with id 425a75dabf8a82d77ed9c50f6a18e36d084afa707cbc78c1ec12885611190077 Jul 27 19:41:12.990841 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:12.990819 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756"] Jul 27 19:41:12.992495 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:41:12.992467 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88007507_fc7d_4645_973d_2eb422dabc19.slice/crio-1628950647b99ed8a135363bc8385fd0b092ad322c9dd100bbe2736584a72c0e WatchSource:0}: Error finding container 1628950647b99ed8a135363bc8385fd0b092ad322c9dd100bbe2736584a72c0e: Status 404 returned error can't find the container with id 1628950647b99ed8a135363bc8385fd0b092ad322c9dd100bbe2736584a72c0e Jul 27 19:41:13.001196 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.001130 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:41:13.005315 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:41:13.005277 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6748484_10f2_43e4_a707_36f77618e88a.slice/crio-e03cf51dc437351971c5b7072a8649ee20babad62e8fd3f928f3df6ccc3806a9 WatchSource:0}: Error finding container e03cf51dc437351971c5b7072a8649ee20babad62e8fd3f928f3df6ccc3806a9: Status 404 returned error can't find the container with id e03cf51dc437351971c5b7072a8649ee20babad62e8fd3f928f3df6ccc3806a9 Jul 27 19:41:13.118494 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.118459 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d975c7cc9-gx82d"] Jul 27 19:41:13.122150 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.122132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.125478 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.125452 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 27 19:41:13.125711 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.125693 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 27 19:41:13.125954 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.125938 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 27 19:41:13.126085 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.126067 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eofh0tvla39kd\"" Jul 27 19:41:13.126142 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.126101 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 27 19:41:13.126958 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.126940 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9wfqj\"" Jul 27 19:41:13.127103 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.126940 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 27 19:41:13.141325 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.141255 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d975c7cc9-gx82d"] Jul 27 19:41:13.253153 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.253077 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.253153 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.253125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.253371 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.253204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-grpc-tls\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.253371 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.253260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.253371 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.253325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.253371 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.253355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck5hp\" (UniqueName: \"kubernetes.io/projected/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-kube-api-access-ck5hp\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.253537 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.253402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-tls\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.253537 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.253494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-metrics-client-ca\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.353680 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.353645 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" event={"ID":"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8","Type":"ContainerStarted","Data":"425a75dabf8a82d77ed9c50f6a18e36d084afa707cbc78c1ec12885611190077"} Jul 27 19:41:13.353900 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.353876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.353973 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.353914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck5hp\" (UniqueName: \"kubernetes.io/projected/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-kube-api-access-ck5hp\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.353973 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.353955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-tls\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.354135 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.353999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-metrics-client-ca\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.354135 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.354067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.354135 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.354098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.354135 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.354130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-grpc-tls\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.354397 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.354157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.355621 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.355590 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rk58b" event={"ID":"37dda389-7aa6-461e-8fcb-9d00ae6dde05","Type":"ContainerStarted","Data":"28593a9c9f469f134b384a52b7855126d3ff7cab07c6ed704e32ad848ae09188"} Jul 27 19:41:13.356377 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.356346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-metrics-client-ca\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.356637 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.356616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.357221 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.357168 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.357340 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.357251 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.357435 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.357409 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.357490 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.357465 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-thanos-querier-tls\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.358040 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.358020 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-secret-grpc-tls\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.358837 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.358749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" event={"ID":"88007507-fc7d-4645-973d-2eb422dabc19","Type":"ContainerStarted","Data":"5476acf899fdbe6a12d217c9f9942f84456458007c88a87fc9d644dd8d03752f"} Jul 27 19:41:13.358837 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.358802 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" event={"ID":"88007507-fc7d-4645-973d-2eb422dabc19","Type":"ContainerStarted","Data":"fdb30ced311ff7827819e207bd96515fe125133d8dbed09be151b29f70601739"} Jul 27 19:41:13.358837 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.358832 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" event={"ID":"88007507-fc7d-4645-973d-2eb422dabc19","Type":"ContainerStarted","Data":"1628950647b99ed8a135363bc8385fd0b092ad322c9dd100bbe2736584a72c0e"} Jul 27 19:41:13.361019 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.360996 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzhgg" event={"ID":"baa95d10-67aa-49d1-8e70-a7d0e2652e95","Type":"ContainerStarted","Data":"ee040435cbb16196784c67f43ec17bf5a4249595a1c759939d5aff3dafedb69e"} Jul 27 19:41:13.362741 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.362715 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerStarted","Data":"e03cf51dc437351971c5b7072a8649ee20babad62e8fd3f928f3df6ccc3806a9"} Jul 27 19:41:13.367904 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.367875 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck5hp\" (UniqueName: \"kubernetes.io/projected/cb4a5e90-1688-4ab4-b788-1a14309cd7b9-kube-api-access-ck5hp\") pod \"thanos-querier-6d975c7cc9-gx82d\" (UID: \"cb4a5e90-1688-4ab4-b788-1a14309cd7b9\") " pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.381478 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.381425 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tzhgg" podStartSLOduration=1.688871311 podStartE2EDuration="20.381403267s" totalImagesPullingTime="18.692531956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:40:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:13.379922218 +0000 UTC m=+77.992506826" watchObservedRunningTime="2026-07-27 19:41:13.381403267 +0000 UTC m=+77.993987851" Jul 27 19:41:13.434021 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.433992 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:13.600262 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:13.600124 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d975c7cc9-gx82d"] Jul 27 19:41:14.138032 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.137994 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:14.235905 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.235870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:14.336858 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.336828 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:14.369515 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.369480 2573 generic.go:356] "Generic (PLEG): container finished" podID="37dda389-7aa6-461e-8fcb-9d00ae6dde05" containerID="9daa82b470d86e9fd2728a6fad64e939bf32b4395baa38362598f466b21f0d0f" exitCode=0 Jul 27 19:41:14.369697 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.369569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rk58b" event={"ID":"37dda389-7aa6-461e-8fcb-9d00ae6dde05","Type":"ContainerDied","Data":"9daa82b470d86e9fd2728a6fad64e939bf32b4395baa38362598f466b21f0d0f"} Jul 27 19:41:14.370993 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.370963 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" event={"ID":"cb4a5e90-1688-4ab4-b788-1a14309cd7b9","Type":"ContainerStarted","Data":"5c7aa54b394baaf8a9ed8722d54611c0ba5775f0662b3c87a6578f988becc0c6"} Jul 27 19:41:14.401034 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.400938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:14.626436 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.626396 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:14.626621 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.626468 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:14.632255 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:14.632223 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:15.383860 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.383769 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" event={"ID":"88007507-fc7d-4645-973d-2eb422dabc19","Type":"ContainerStarted","Data":"fcedab4325656bc5bef747e805e3a9bab815926962538dab2df57adda674d21f"} Jul 27 19:41:15.385522 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.385490 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6748484-10f2-43e4-a707-36f77618e88a" containerID="82f58b7b046a2d8067e3e7927904055e3acdb285b2502dff3593f08eb8386aaa" exitCode=0 Jul 27 19:41:15.385639 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.385578 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerDied","Data":"82f58b7b046a2d8067e3e7927904055e3acdb285b2502dff3593f08eb8386aaa"} Jul 27 19:41:15.388112 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.388077 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" event={"ID":"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8","Type":"ContainerStarted","Data":"72642a22a93cf88bd95ff967ce194ae463e99bae1ab473bf7a267a0f5c2564df"} Jul 27 19:41:15.388207 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.388136 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" event={"ID":"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8","Type":"ContainerStarted","Data":"feef401b6e7bca96e0caa8a79a7b40f5621e944ab1068fc5e216f43fe785504e"} Jul 27 19:41:15.388207 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.388157 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" event={"ID":"7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8","Type":"ContainerStarted","Data":"f9fee9d8dc6d3ae83eb4d0c62d08203871fb84dcca05be500292c8fdbe587cc3"} Jul 27 19:41:15.390509 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.390488 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rk58b" event={"ID":"37dda389-7aa6-461e-8fcb-9d00ae6dde05","Type":"ContainerStarted","Data":"8e937f9e4a291b04dbbe337897f54ce7989846cb1388dcbe916dd4f46b2a5303"} Jul 27 19:41:15.390643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.390627 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rk58b" event={"ID":"37dda389-7aa6-461e-8fcb-9d00ae6dde05","Type":"ContainerStarted","Data":"018c2f01b8312d877d146eef011b6249522ece179d969aa1a4ffc9f4dd0afed6"} Jul 27 19:41:15.398667 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.398646 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:41:15.403536 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.403499 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tx756" podStartSLOduration=2.702631554 podStartE2EDuration="4.403484299s" totalImagesPullingTime="1.700852745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:15.402009979 +0000 UTC m=+80.014594563" watchObservedRunningTime="2026-07-27 19:41:15.403484299 +0000 UTC m=+80.016068884" Jul 27 19:41:15.423568 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.423513 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zr86l" podStartSLOduration=2.348947311 podStartE2EDuration="4.423494284s" totalImagesPullingTime="2.074546973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:15.422742733 +0000 UTC m=+80.035327379" watchObservedRunningTime="2026-07-27 19:41:15.423494284 +0000 UTC m=+80.036078868" Jul 27 19:41:15.500474 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.500408 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rk58b" podStartSLOduration=3.546282447 podStartE2EDuration="4.500385713s" totalImagesPullingTime="842.959684ms" totalInitContainerRuntime="111.143582ms" isStatefulPod=true podCreationTimestamp="2026-07-27 19:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:15.49993388 +0000 UTC m=+80.112518484" watchObservedRunningTime="2026-07-27 19:41:15.500385713 +0000 UTC m=+80.112970298" Jul 27 19:41:15.553377 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.553343 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fc9b4c4b-5vvhz"] Jul 27 19:41:15.559726 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.559693 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-566b9db6d4-pxqr5"] Jul 27 19:41:15.564362 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.564341 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.567449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.566768 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 27 19:41:15.567449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.566768 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-swxdb\"" Jul 27 19:41:15.567449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.567235 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 27 19:41:15.567449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.567339 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-413k1slvraqhn\"" Jul 27 19:41:15.567726 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.567508 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 27 19:41:15.567726 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.567574 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 27 19:41:15.572856 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.572798 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-566b9db6d4-pxqr5"] Jul 27 19:41:15.677685 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.677589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-secret-metrics-server-client-certs\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.677685 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.677650 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.677913 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.677712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-metrics-server-audit-profiles\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.677913 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.677740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-audit-log\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.677913 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.677795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-secret-metrics-server-tls\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.677913 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.677822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-client-ca-bundle\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.678069 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.677929 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr4nv\" (UniqueName: \"kubernetes.io/projected/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-kube-api-access-tr4nv\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.680936 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.680903 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:15.779390 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.779352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-metrics-server-audit-profiles\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.779390 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.779385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-audit-log\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.779607 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.779430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-secret-metrics-server-tls\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.779607 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.779455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-client-ca-bundle\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.779607 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.779495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr4nv\" (UniqueName: \"kubernetes.io/projected/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-kube-api-access-tr4nv\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.779607 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.779532 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-secret-metrics-server-client-certs\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.779607 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.779570 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.779907 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.779883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-audit-log\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.780516 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.780479 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.780848 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.780822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-metrics-server-audit-profiles\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.780997 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.780975 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:15.782231 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.782208 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-secret-metrics-server-client-certs\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.782333 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.782254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-secret-metrics-server-tls\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.782383 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.782341 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-client-ca-bundle\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.788608 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.788588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr4nv\" (UniqueName: \"kubernetes.io/projected/be0f1e17-ddc8-4d05-895b-4ca6a9504f76-kube-api-access-tr4nv\") pod \"metrics-server-566b9db6d4-pxqr5\" (UID: \"be0f1e17-ddc8-4d05-895b-4ca6a9504f76\") " pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:15.852066 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.852020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:15.879256 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:15.879228 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:16.039899 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:16.039850 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-566b9db6d4-pxqr5"] Jul 27 19:41:16.045317 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:41:16.045280 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0f1e17_ddc8_4d05_895b_4ca6a9504f76.slice/crio-36011cef43da5e5f9fb06ec39698b6e2eaefff6c7e3ceb5ec9f397d396f61f35 WatchSource:0}: Error finding container 36011cef43da5e5f9fb06ec39698b6e2eaefff6c7e3ceb5ec9f397d396f61f35: Status 404 returned error can't find the container with id 36011cef43da5e5f9fb06ec39698b6e2eaefff6c7e3ceb5ec9f397d396f61f35 Jul 27 19:41:16.398428 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:16.398332 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" event={"ID":"cb4a5e90-1688-4ab4-b788-1a14309cd7b9","Type":"ContainerStarted","Data":"695f6a73d414cd961875f4e960bd13e6ea4b18a31d62d3906956fdbb0f98775c"} Jul 27 19:41:16.398885 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:16.398431 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" event={"ID":"cb4a5e90-1688-4ab4-b788-1a14309cd7b9","Type":"ContainerStarted","Data":"3954bb78c4aacaaf7396db4d4868f06cc3f88894663221a63bc241d89877812f"} Jul 27 19:41:16.398885 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:16.398476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" event={"ID":"cb4a5e90-1688-4ab4-b788-1a14309cd7b9","Type":"ContainerStarted","Data":"af1909a13418c58bc8246e1dc8fbe32cf2a791dc5f16f45feafa576c2e34da67"} Jul 27 19:41:16.399931 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:16.399883 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" event={"ID":"be0f1e17-ddc8-4d05-895b-4ca6a9504f76","Type":"ContainerStarted","Data":"36011cef43da5e5f9fb06ec39698b6e2eaefff6c7e3ceb5ec9f397d396f61f35"} Jul 27 19:41:16.501745 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:16.501711 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:16.601973 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:16.601943 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:16.723456 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:16.723422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:16.836812 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:16.836777 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:17.411169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:17.411069 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:17.476913 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:17.476880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:18.062737 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.062711 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:41:18.244639 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.244528 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:18.345243 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.345195 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:18.414490 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.414457 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" event={"ID":"cb4a5e90-1688-4ab4-b788-1a14309cd7b9","Type":"ContainerStarted","Data":"f81fdee68a02a259d07c1134c97cc68040fb19ed42ca04c5d99afc2306329f20"} Jul 27 19:41:18.414948 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.414538 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" event={"ID":"cb4a5e90-1688-4ab4-b788-1a14309cd7b9","Type":"ContainerStarted","Data":"f4a3fd8346da301fbe30e6ddb9c54126eba32e5714abb3df54c90873f0087516"} Jul 27 19:41:18.414948 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.414577 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" event={"ID":"cb4a5e90-1688-4ab4-b788-1a14309cd7b9","Type":"ContainerStarted","Data":"5321adb8edf3884e1735e5e79822d71e6f89c392243c5269e24c0aefd5ed46fb"} Jul 27 19:41:18.414948 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.414624 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:18.416079 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.416056 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" event={"ID":"be0f1e17-ddc8-4d05-895b-4ca6a9504f76","Type":"ContainerStarted","Data":"403ac922b7b4e7a69141fd483f7cde80c4fbf1171d4f9c31cd255ed161596de3"} Jul 27 19:41:18.419406 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.419383 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerStarted","Data":"979ef6b9a6f5077c3664bc0c8196e4db3e1db4e87ec86a27a8e44ad42bd0c62a"} Jul 27 19:41:18.419479 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.419447 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerStarted","Data":"7871b69081198dd79af4e4e35daed48b28e3659fd059e27314a7526e64def5c5"} Jul 27 19:41:18.419516 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.419486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerStarted","Data":"b46563cc8d1e68e29765d6712c04e2699fc288c0e562401e50c41670da18aa89"} Jul 27 19:41:18.419555 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.419524 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerStarted","Data":"3ebaf84b942c1bab551b649688c334e6bfbbe4b17fb661b551efbbe14675b698"} Jul 27 19:41:18.419598 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.419567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerStarted","Data":"6c182519f42c627fe1cb114c402f04dd772a18888f9308a1d088531520036d40"} Jul 27 19:41:18.419640 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.419607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerStarted","Data":"5deebd4145712e4a93be918a46a4a7d014885d956a47a6c385366a6869be9b66"} Jul 27 19:41:18.439082 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.439031 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" podStartSLOduration=1.668464301 podStartE2EDuration="5.439014577s" totalImagesPullingTime="3.770550276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:18.437905258 +0000 UTC m=+83.050489845" watchObservedRunningTime="2026-07-27 19:41:18.439014577 +0000 UTC m=+83.051599159" Jul 27 19:41:18.468621 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.468551 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.375742205 podStartE2EDuration="6.468534322s" totalImagesPullingTime="4.063971554s" totalInitContainerRuntime="28.820563ms" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:18.46590268 +0000 UTC m=+83.078487291" watchObservedRunningTime="2026-07-27 19:41:18.468534322 +0000 UTC m=+83.081118904" Jul 27 19:41:18.487926 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:18.487859 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" podStartSLOduration=1.848416998 podStartE2EDuration="3.487842133s" totalImagesPullingTime="1.639425135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:18.486361545 +0000 UTC m=+83.098946130" watchObservedRunningTime="2026-07-27 19:41:18.487842133 +0000 UTC m=+83.100426717" Jul 27 19:41:21.133586 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:21.133547 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:21.190424 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:21.190387 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:21.438117 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:21.438080 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:21.538591 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:21.538559 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:24.429912 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:24.429886 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d975c7cc9-gx82d" Jul 27 19:41:28.279688 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.279657 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5867c6bcb5-nzb4g"] Jul 27 19:41:28.284194 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.284176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.292274 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.292251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-oauth-serving-cert\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.292377 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.292279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-serving-cert\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.292377 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.292310 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9c5w\" (UniqueName: \"kubernetes.io/projected/61a1b4f0-9be0-4863-acff-dbce02ecfb90-kube-api-access-g9c5w\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.292455 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.292394 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-trusted-ca-bundle\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.292455 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.292421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-config\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.292455 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.292442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-service-ca\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.292563 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.292494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-oauth-config\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.297355 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.297335 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5867c6bcb5-nzb4g"] Jul 27 19:41:28.392859 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.392822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-trusted-ca-bundle\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.392859 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.392858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-config\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.393118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.392880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-service-ca\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.393118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.392961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-oauth-config\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.393118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.393056 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-oauth-serving-cert\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.393118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.393088 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-serving-cert\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.393118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.393114 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9c5w\" (UniqueName: \"kubernetes.io/projected/61a1b4f0-9be0-4863-acff-dbce02ecfb90-kube-api-access-g9c5w\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.393709 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.393682 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-service-ca\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.393824 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.393713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-oauth-serving-cert\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.393824 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.393762 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-config\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.393911 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.393886 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-trusted-ca-bundle\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.395842 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.395820 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-oauth-config\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.395885 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.395837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-serving-cert\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.412004 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.411975 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9c5w\" (UniqueName: \"kubernetes.io/projected/61a1b4f0-9be0-4863-acff-dbce02ecfb90-kube-api-access-g9c5w\") pod \"console-5867c6bcb5-nzb4g\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.593471 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.593375 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:28.724735 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:28.724710 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5867c6bcb5-nzb4g"] Jul 27 19:41:28.727175 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:41:28.727150 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a1b4f0_9be0_4863_acff_dbce02ecfb90.slice/crio-0cdc5cbf9d32be83e968e1f724dc8549b08f470da8e5696ffd668aaa89260c16 WatchSource:0}: Error finding container 0cdc5cbf9d32be83e968e1f724dc8549b08f470da8e5696ffd668aaa89260c16: Status 404 returned error can't find the container with id 0cdc5cbf9d32be83e968e1f724dc8549b08f470da8e5696ffd668aaa89260c16 Jul 27 19:41:29.467688 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:29.467655 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5867c6bcb5-nzb4g" event={"ID":"61a1b4f0-9be0-4863-acff-dbce02ecfb90","Type":"ContainerStarted","Data":"6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759"} Jul 27 19:41:29.468087 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:29.467702 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5867c6bcb5-nzb4g" event={"ID":"61a1b4f0-9be0-4863-acff-dbce02ecfb90","Type":"ContainerStarted","Data":"0cdc5cbf9d32be83e968e1f724dc8549b08f470da8e5696ffd668aaa89260c16"} Jul 27 19:41:29.489322 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:29.489247 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5867c6bcb5-nzb4g" podStartSLOduration=1.489227533 podStartE2EDuration="1.489227533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:29.488716128 +0000 UTC m=+94.101300727" watchObservedRunningTime="2026-07-27 19:41:29.489227533 +0000 UTC m=+94.101812096" Jul 27 19:41:32.391648 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:32.391615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:32.491542 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:32.491510 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:35.880394 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:35.880365 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:35.880825 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:35.880449 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:38.594466 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:38.594428 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:38.594859 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:38.594511 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:38.599073 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:38.599054 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:39.510334 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:39.510287 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:41:39.559381 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:39.559345 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c8898f6bb-5f57l"] Jul 27 19:41:39.697097 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:39.697069 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:39.797398 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:39.797326 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:39.860076 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:39.860040 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:40.340321 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:40.340277 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-md4vm" Jul 27 19:41:40.581458 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:40.581413 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66fc9b4c4b-5vvhz" podUID="375dc3d2-f175-41b4-81c1-29e98795cd7a" containerName="console" containerID="cri-o://7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367" gracePeriod=15 Jul 27 19:41:41.355124 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.355092 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:41:41.407989 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.407959 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-oauth-config\") pod \"375dc3d2-f175-41b4-81c1-29e98795cd7a\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " Jul 27 19:41:41.408164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.408013 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnsw5\" (UniqueName: \"kubernetes.io/projected/375dc3d2-f175-41b4-81c1-29e98795cd7a-kube-api-access-tnsw5\") pod \"375dc3d2-f175-41b4-81c1-29e98795cd7a\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " Jul 27 19:41:41.408164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.408041 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-serving-cert\") pod \"375dc3d2-f175-41b4-81c1-29e98795cd7a\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " Jul 27 19:41:41.408164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.408057 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-config\") pod \"375dc3d2-f175-41b4-81c1-29e98795cd7a\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " Jul 27 19:41:41.408164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.408078 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-service-ca\") pod \"375dc3d2-f175-41b4-81c1-29e98795cd7a\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " Jul 27 19:41:41.408164 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.408106 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-oauth-serving-cert\") pod \"375dc3d2-f175-41b4-81c1-29e98795cd7a\" (UID: \"375dc3d2-f175-41b4-81c1-29e98795cd7a\") " Jul 27 19:41:41.408548 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.408519 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-service-ca" (OuterVolumeSpecName: "service-ca") pod "375dc3d2-f175-41b4-81c1-29e98795cd7a" (UID: "375dc3d2-f175-41b4-81c1-29e98795cd7a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:41:41.408627 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.408524 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-config" (OuterVolumeSpecName: "console-config") pod "375dc3d2-f175-41b4-81c1-29e98795cd7a" (UID: "375dc3d2-f175-41b4-81c1-29e98795cd7a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:41:41.408627 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.408578 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "375dc3d2-f175-41b4-81c1-29e98795cd7a" (UID: "375dc3d2-f175-41b4-81c1-29e98795cd7a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:41:41.410599 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.410572 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "375dc3d2-f175-41b4-81c1-29e98795cd7a" (UID: "375dc3d2-f175-41b4-81c1-29e98795cd7a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:41:41.410697 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.410618 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "375dc3d2-f175-41b4-81c1-29e98795cd7a" (UID: "375dc3d2-f175-41b4-81c1-29e98795cd7a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:41:41.410697 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.410616 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375dc3d2-f175-41b4-81c1-29e98795cd7a-kube-api-access-tnsw5" (OuterVolumeSpecName: "kube-api-access-tnsw5") pod "375dc3d2-f175-41b4-81c1-29e98795cd7a" (UID: "375dc3d2-f175-41b4-81c1-29e98795cd7a"). InnerVolumeSpecName "kube-api-access-tnsw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:41:41.509118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.509046 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:41:41.509118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.509074 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-oauth-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:41:41.509118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.509084 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-oauth-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:41:41.509118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.509093 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnsw5\" (UniqueName: \"kubernetes.io/projected/375dc3d2-f175-41b4-81c1-29e98795cd7a-kube-api-access-tnsw5\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:41:41.509118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.509102 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:41:41.509118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.509109 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/375dc3d2-f175-41b4-81c1-29e98795cd7a-console-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:41:41.515251 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.515219 2573 generic.go:356] "Generic (PLEG): container finished" podID="375dc3d2-f175-41b4-81c1-29e98795cd7a" containerID="7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367" exitCode=0 Jul 27 19:41:41.515384 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.515269 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fc9b4c4b-5vvhz" Jul 27 19:41:41.515384 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.515323 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fc9b4c4b-5vvhz" event={"ID":"375dc3d2-f175-41b4-81c1-29e98795cd7a","Type":"ContainerDied","Data":"7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367"} Jul 27 19:41:41.515384 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.515371 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fc9b4c4b-5vvhz" event={"ID":"375dc3d2-f175-41b4-81c1-29e98795cd7a","Type":"ContainerDied","Data":"2bbfa819b949f4551322cc76717e870cef35037ff6cdb54cd95da698e7091822"} Jul 27 19:41:41.515485 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.515392 2573 scope.go:117] "RemoveContainer" containerID="7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367" Jul 27 19:41:41.523796 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.523780 2573 scope.go:117] "RemoveContainer" containerID="7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367" Jul 27 19:41:41.524051 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:41:41.524034 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367\": container with ID starting with 7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367 not found: ID does not exist" containerID="7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367" Jul 27 19:41:41.524101 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.524058 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367"} err="failed to get container status \"7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367\": rpc error: code = NotFound desc = could not find container \"7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367\": container with ID starting with 7a944a46c2702016048588bb0ad9c1b201d200ffcbc7441ec22119092bb8e367 not found: ID does not exist" Jul 27 19:41:41.535980 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.535961 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fc9b4c4b-5vvhz"] Jul 27 19:41:41.537992 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.537973 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66fc9b4c4b-5vvhz"] Jul 27 19:41:41.877962 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:41.875779 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="375dc3d2-f175-41b4-81c1-29e98795cd7a" path="/var/lib/kubelet/pods/375dc3d2-f175-41b4-81c1-29e98795cd7a/volumes" Jul 27 19:41:44.139787 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:44.139747 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:44.231716 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:44.231676 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:55.884910 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:55.884874 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:55.888644 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:55.888620 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-566b9db6d4-pxqr5" Jul 27 19:41:56.888776 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:56.888736 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:41:56.949204 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:41:56.949166 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:02.649916 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:02.649876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:02.707236 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:02.707209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:04.579055 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.579016 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c8898f6bb-5f57l" podUID="09de69fa-ce00-4850-a64f-3e11ebdb0f16" containerName="console" containerID="cri-o://61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d" gracePeriod=15 Jul 27 19:42:04.818247 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.818221 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:42:04.903344 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903246 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r27c8\" (UniqueName: \"kubernetes.io/projected/09de69fa-ce00-4850-a64f-3e11ebdb0f16-kube-api-access-r27c8\") pod \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " Jul 27 19:42:04.903496 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903345 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-oauth-serving-cert\") pod \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " Jul 27 19:42:04.903496 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903401 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-oauth-config\") pod \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " Jul 27 19:42:04.903496 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903428 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-trusted-ca-bundle\") pod \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " Jul 27 19:42:04.903496 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903457 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-service-ca\") pod \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " Jul 27 19:42:04.903688 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903501 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-config\") pod \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " Jul 27 19:42:04.903688 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903544 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-serving-cert\") pod \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\" (UID: \"09de69fa-ce00-4850-a64f-3e11ebdb0f16\") " Jul 27 19:42:04.903835 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903801 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "09de69fa-ce00-4850-a64f-3e11ebdb0f16" (UID: "09de69fa-ce00-4850-a64f-3e11ebdb0f16"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:42:04.903892 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903827 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "09de69fa-ce00-4850-a64f-3e11ebdb0f16" (UID: "09de69fa-ce00-4850-a64f-3e11ebdb0f16"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:42:04.903892 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903861 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-config" (OuterVolumeSpecName: "console-config") pod "09de69fa-ce00-4850-a64f-3e11ebdb0f16" (UID: "09de69fa-ce00-4850-a64f-3e11ebdb0f16"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:42:04.903968 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.903893 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-service-ca" (OuterVolumeSpecName: "service-ca") pod "09de69fa-ce00-4850-a64f-3e11ebdb0f16" (UID: "09de69fa-ce00-4850-a64f-3e11ebdb0f16"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:42:04.905707 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.905685 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "09de69fa-ce00-4850-a64f-3e11ebdb0f16" (UID: "09de69fa-ce00-4850-a64f-3e11ebdb0f16"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:42:04.905824 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.905729 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09de69fa-ce00-4850-a64f-3e11ebdb0f16-kube-api-access-r27c8" (OuterVolumeSpecName: "kube-api-access-r27c8") pod "09de69fa-ce00-4850-a64f-3e11ebdb0f16" (UID: "09de69fa-ce00-4850-a64f-3e11ebdb0f16"). InnerVolumeSpecName "kube-api-access-r27c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:42:04.905824 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:04.905756 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "09de69fa-ce00-4850-a64f-3e11ebdb0f16" (UID: "09de69fa-ce00-4850-a64f-3e11ebdb0f16"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:42:05.004836 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.004796 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r27c8\" (UniqueName: \"kubernetes.io/projected/09de69fa-ce00-4850-a64f-3e11ebdb0f16-kube-api-access-r27c8\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:05.004836 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.004828 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-oauth-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:05.004836 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.004838 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-oauth-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:05.004836 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.004846 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-trusted-ca-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:05.005102 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.004858 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:05.005102 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.004866 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:05.005102 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.004874 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09de69fa-ce00-4850-a64f-3e11ebdb0f16-console-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:05.608866 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.608832 2573 generic.go:356] "Generic (PLEG): container finished" podID="09de69fa-ce00-4850-a64f-3e11ebdb0f16" containerID="61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d" exitCode=0 Jul 27 19:42:05.609330 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.608885 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8898f6bb-5f57l" Jul 27 19:42:05.609330 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.608926 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8898f6bb-5f57l" event={"ID":"09de69fa-ce00-4850-a64f-3e11ebdb0f16","Type":"ContainerDied","Data":"61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d"} Jul 27 19:42:05.609330 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.608979 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8898f6bb-5f57l" event={"ID":"09de69fa-ce00-4850-a64f-3e11ebdb0f16","Type":"ContainerDied","Data":"865dff5664575020dc25510a4d19271b05c63c854b65d0bc03c5d25e77ab561d"} Jul 27 19:42:05.609330 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.609007 2573 scope.go:117] "RemoveContainer" containerID="61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d" Jul 27 19:42:05.617479 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.617463 2573 scope.go:117] "RemoveContainer" containerID="61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d" Jul 27 19:42:05.617724 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:42:05.617704 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d\": container with ID starting with 61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d not found: ID does not exist" containerID="61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d" Jul 27 19:42:05.617786 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.617732 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d"} err="failed to get container status \"61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d\": rpc error: code = NotFound desc = could not find container \"61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d\": container with ID starting with 61136d8d0248df6fe3f5d9fe88b3f10963d81dcb19e669bae3330e91be7c743d not found: ID does not exist" Jul 27 19:42:05.630106 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.630081 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c8898f6bb-5f57l"] Jul 27 19:42:05.635081 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.635059 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c8898f6bb-5f57l"] Jul 27 19:42:05.878480 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:05.878404 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09de69fa-ce00-4850-a64f-3e11ebdb0f16" path="/var/lib/kubelet/pods/09de69fa-ce00-4850-a64f-3e11ebdb0f16/volumes" Jul 27 19:42:27.300937 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:27.300906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:27.357771 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:27.357737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:31.091591 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.091544 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:31.574093 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.574060 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:42:31.574458 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.574427 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="alertmanager" containerID="cri-o://5deebd4145712e4a93be918a46a4a7d014885d956a47a6c385366a6869be9b66" gracePeriod=120 Jul 27 19:42:31.574541 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.574464 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy-metric" containerID="cri-o://7871b69081198dd79af4e4e35daed48b28e3659fd059e27314a7526e64def5c5" gracePeriod=120 Jul 27 19:42:31.574597 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.574525 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="prom-label-proxy" containerID="cri-o://979ef6b9a6f5077c3664bc0c8196e4db3e1db4e87ec86a27a8e44ad42bd0c62a" gracePeriod=120 Jul 27 19:42:31.574597 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.574533 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy-web" containerID="cri-o://3ebaf84b942c1bab551b649688c334e6bfbbe4b17fb661b551efbbe14675b698" gracePeriod=120 Jul 27 19:42:31.574597 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.574589 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="config-reloader" containerID="cri-o://6c182519f42c627fe1cb114c402f04dd772a18888f9308a1d088531520036d40" gracePeriod=120 Jul 27 19:42:31.574739 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.574563 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy" containerID="cri-o://b46563cc8d1e68e29765d6712c04e2699fc288c0e562401e50c41670da18aa89" gracePeriod=120 Jul 27 19:42:31.708265 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.708238 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6748484-10f2-43e4-a707-36f77618e88a" containerID="979ef6b9a6f5077c3664bc0c8196e4db3e1db4e87ec86a27a8e44ad42bd0c62a" exitCode=0 Jul 27 19:42:31.708265 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.708264 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6748484-10f2-43e4-a707-36f77618e88a" containerID="b46563cc8d1e68e29765d6712c04e2699fc288c0e562401e50c41670da18aa89" exitCode=0 Jul 27 19:42:31.708265 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.708270 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6748484-10f2-43e4-a707-36f77618e88a" containerID="6c182519f42c627fe1cb114c402f04dd772a18888f9308a1d088531520036d40" exitCode=0 Jul 27 19:42:31.708489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.708275 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6748484-10f2-43e4-a707-36f77618e88a" containerID="5deebd4145712e4a93be918a46a4a7d014885d956a47a6c385366a6869be9b66" exitCode=0 Jul 27 19:42:31.708489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.708337 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerDied","Data":"979ef6b9a6f5077c3664bc0c8196e4db3e1db4e87ec86a27a8e44ad42bd0c62a"} Jul 27 19:42:31.708489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.708421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerDied","Data":"b46563cc8d1e68e29765d6712c04e2699fc288c0e562401e50c41670da18aa89"} Jul 27 19:42:31.708489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.708466 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerDied","Data":"6c182519f42c627fe1cb114c402f04dd772a18888f9308a1d088531520036d40"} Jul 27 19:42:31.708639 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:31.708505 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerDied","Data":"5deebd4145712e4a93be918a46a4a7d014885d956a47a6c385366a6869be9b66"} Jul 27 19:42:32.715920 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.715883 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6748484-10f2-43e4-a707-36f77618e88a" containerID="7871b69081198dd79af4e4e35daed48b28e3659fd059e27314a7526e64def5c5" exitCode=0 Jul 27 19:42:32.715920 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.715916 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6748484-10f2-43e4-a707-36f77618e88a" containerID="3ebaf84b942c1bab551b649688c334e6bfbbe4b17fb661b551efbbe14675b698" exitCode=0 Jul 27 19:42:32.716447 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.715952 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerDied","Data":"7871b69081198dd79af4e4e35daed48b28e3659fd059e27314a7526e64def5c5"} Jul 27 19:42:32.716447 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.716077 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerDied","Data":"3ebaf84b942c1bab551b649688c334e6bfbbe4b17fb661b551efbbe14675b698"} Jul 27 19:42:32.812025 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.812003 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:32.922473 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922403 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-web-config\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922473 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922443 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-main-tls\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922473 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922460 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-trusted-ca-bundle\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922720 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922488 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-config-out\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922720 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922503 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-tls-assets\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922720 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922527 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-config-volume\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922720 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922561 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-main-db\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922720 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922598 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922720 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922628 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-metrics-client-ca\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922720 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922656 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.922720 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922712 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-cluster-tls-config\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.923104 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922742 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-web\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.923104 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922773 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kptb4\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-kube-api-access-kptb4\") pod \"f6748484-10f2-43e4-a707-36f77618e88a\" (UID: \"f6748484-10f2-43e4-a707-36f77618e88a\") " Jul 27 19:42:32.923104 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.922868 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:42:32.923345 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.923316 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:42:32.923433 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.923053 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:32.924144 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.923696 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:42:32.925386 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.925357 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:42:32.925753 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.925729 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-config-out" (OuterVolumeSpecName: "config-out") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:42:32.926385 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.926339 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:42:32.926476 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.926374 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-config-volume" (OuterVolumeSpecName: "config-volume") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:42:32.926476 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.926399 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:42:32.926476 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.926431 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-kube-api-access-kptb4" (OuterVolumeSpecName: "kube-api-access-kptb4") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "kube-api-access-kptb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:42:32.926711 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.926685 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:42:32.927585 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.927562 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:42:32.930630 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.930586 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:42:32.936415 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:32.936393 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-web-config" (OuterVolumeSpecName: "web-config") pod "f6748484-10f2-43e4-a707-36f77618e88a" (UID: "f6748484-10f2-43e4-a707-36f77618e88a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:42:33.023903 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023878 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.023903 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023900 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kptb4\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-kube-api-access-kptb4\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023910 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-web-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023919 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-main-tls\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023928 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-config-out\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023935 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6748484-10f2-43e4-a707-36f77618e88a-tls-assets\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023942 2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023950 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6748484-10f2-43e4-a707-36f77618e88a-alertmanager-main-db\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023958 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023966 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6748484-10f2-43e4-a707-36f77618e88a-metrics-client-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023974 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.024027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.023983 2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6748484-10f2-43e4-a707-36f77618e88a-cluster-tls-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:42:33.722676 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.722642 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6748484-10f2-43e4-a707-36f77618e88a","Type":"ContainerDied","Data":"e03cf51dc437351971c5b7072a8649ee20babad62e8fd3f928f3df6ccc3806a9"} Jul 27 19:42:33.723042 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.722720 2573 scope.go:117] "RemoveContainer" containerID="979ef6b9a6f5077c3664bc0c8196e4db3e1db4e87ec86a27a8e44ad42bd0c62a" Jul 27 19:42:33.723042 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.722659 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.730512 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.730490 2573 scope.go:117] "RemoveContainer" containerID="7871b69081198dd79af4e4e35daed48b28e3659fd059e27314a7526e64def5c5" Jul 27 19:42:33.737270 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.737254 2573 scope.go:117] "RemoveContainer" containerID="b46563cc8d1e68e29765d6712c04e2699fc288c0e562401e50c41670da18aa89" Jul 27 19:42:33.743747 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.743733 2573 scope.go:117] "RemoveContainer" containerID="3ebaf84b942c1bab551b649688c334e6bfbbe4b17fb661b551efbbe14675b698" Jul 27 19:42:33.746356 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.746334 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:42:33.750023 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.750003 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:42:33.751252 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.751236 2573 scope.go:117] "RemoveContainer" containerID="6c182519f42c627fe1cb114c402f04dd772a18888f9308a1d088531520036d40" Jul 27 19:42:33.757924 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.757907 2573 scope.go:117] "RemoveContainer" containerID="5deebd4145712e4a93be918a46a4a7d014885d956a47a6c385366a6869be9b66" Jul 27 19:42:33.764535 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.764504 2573 scope.go:117] "RemoveContainer" containerID="82f58b7b046a2d8067e3e7927904055e3acdb285b2502dff3593f08eb8386aaa" Jul 27 19:42:33.782110 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782089 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:42:33.782377 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782364 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="config-reloader" Jul 27 19:42:33.782430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782378 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="config-reloader" Jul 27 19:42:33.782430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782388 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy-metric" Jul 27 19:42:33.782430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782394 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy-metric" Jul 27 19:42:33.782430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782406 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="init-config-reloader" Jul 27 19:42:33.782430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782413 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="init-config-reloader" Jul 27 19:42:33.782430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782420 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09de69fa-ce00-4850-a64f-3e11ebdb0f16" containerName="console" Jul 27 19:42:33.782430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782424 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="09de69fa-ce00-4850-a64f-3e11ebdb0f16" containerName="console" Jul 27 19:42:33.782430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782431 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="prom-label-proxy" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782436 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="prom-label-proxy" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782444 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="375dc3d2-f175-41b4-81c1-29e98795cd7a" containerName="console" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782449 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="375dc3d2-f175-41b4-81c1-29e98795cd7a" containerName="console" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782454 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy-web" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782459 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy-web" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782465 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782470 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782478 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="alertmanager" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782484 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="alertmanager" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782530 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782538 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="alertmanager" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782545 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy-web" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782551 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="kube-rbac-proxy-metric" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782556 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="375dc3d2-f175-41b4-81c1-29e98795cd7a" containerName="console" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782562 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="config-reloader" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782567 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="09de69fa-ce00-4850-a64f-3e11ebdb0f16" containerName="console" Jul 27 19:42:33.782682 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.782573 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6748484-10f2-43e4-a707-36f77618e88a" containerName="prom-label-proxy" Jul 27 19:42:33.786345 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.786326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.788862 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.788838 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 27 19:42:33.788862 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.788849 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 27 19:42:33.789020 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.788838 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 27 19:42:33.789020 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.788843 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 27 19:42:33.789020 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.788957 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 27 19:42:33.789020 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.789005 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 27 19:42:33.789362 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.789345 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 27 19:42:33.789413 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.789397 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jws6n\"" Jul 27 19:42:33.789451 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.789417 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 27 19:42:33.794611 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.794595 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 27 19:42:33.801590 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.801571 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:42:33.831318 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bfpl\" (UniqueName: \"kubernetes.io/projected/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-kube-api-access-4bfpl\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831477 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831336 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831477 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831477 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831477 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831477 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831641 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831480 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-web-config\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831641 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831641 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831641 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831585 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831641 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831787 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-config-out\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.831787 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.831679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.874030 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.873952 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6748484-10f2-43e4-a707-36f77618e88a" path="/var/lib/kubelet/pods/f6748484-10f2-43e4-a707-36f77618e88a/volumes" Jul 27 19:42:33.932247 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-config-out\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932427 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932427 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bfpl\" (UniqueName: \"kubernetes.io/projected/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-kube-api-access-4bfpl\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932534 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932432 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932534 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932534 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932680 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932680 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932680 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-web-config\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932680 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932873 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932873 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.932873 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.932784 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.933853 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.933830 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.935823 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.935673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.935917 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.935825 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.935917 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.935882 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.936944 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.936913 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-web-config\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.937117 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.937095 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.937177 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.937163 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-config-out\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.937418 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.937397 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.937589 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.937555 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.939887 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.937719 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.940859 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.940812 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.942362 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.940911 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:33.943286 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:33.943267 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bfpl\" (UniqueName: \"kubernetes.io/projected/1f849de3-35a1-40b9-a7b1-817e1ea3a0f0-kube-api-access-4bfpl\") pod \"alertmanager-main-0\" (UID: \"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:34.095796 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:34.095759 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:42:34.247266 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:34.247235 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:42:34.250739 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:42:34.250716 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f849de3_35a1_40b9_a7b1_817e1ea3a0f0.slice/crio-8fa099f71a038e46bcda70ecf65e957887ec0e5e64dbec1ef0bd4838989a99ac WatchSource:0}: Error finding container 8fa099f71a038e46bcda70ecf65e957887ec0e5e64dbec1ef0bd4838989a99ac: Status 404 returned error can't find the container with id 8fa099f71a038e46bcda70ecf65e957887ec0e5e64dbec1ef0bd4838989a99ac Jul 27 19:42:34.727746 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:34.727713 2573 generic.go:356] "Generic (PLEG): container finished" podID="1f849de3-35a1-40b9-a7b1-817e1ea3a0f0" containerID="bfc613675d10def4af93811a5b8991b21186b8a3f300c7c6a1b50d8c2d85b18e" exitCode=0 Jul 27 19:42:34.728150 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:34.727861 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0","Type":"ContainerDied","Data":"bfc613675d10def4af93811a5b8991b21186b8a3f300c7c6a1b50d8c2d85b18e"} Jul 27 19:42:34.728150 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:34.727945 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0","Type":"ContainerStarted","Data":"8fa099f71a038e46bcda70ecf65e957887ec0e5e64dbec1ef0bd4838989a99ac"} Jul 27 19:42:35.737780 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:35.737741 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0","Type":"ContainerStarted","Data":"f74fcf8837bf63b9be2e9ef53a5bd850d0c98e05d599b10aeb504be999a75bb5"} Jul 27 19:42:35.738253 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:35.737816 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0","Type":"ContainerStarted","Data":"c99a2b26dc69aef3f1a6ce4210c6ec0a4a3fa3483394539c10d4f0f752647ac0"} Jul 27 19:42:35.738253 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:35.737856 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0","Type":"ContainerStarted","Data":"690aa7ea56e40928a03345d6186e93aa5564825ac13f35e7b35206937d61554c"} Jul 27 19:42:35.738253 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:35.737898 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0","Type":"ContainerStarted","Data":"bd4d76d5b42b043ad86624e92d510a0246fab2f5244aecffed3f9647f9e10cc7"} Jul 27 19:42:35.738253 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:35.737942 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0","Type":"ContainerStarted","Data":"6b7ebf3a0d733e0f107a84a81204d53c9f2baeab85ed2d44d289d4c2c6c519a0"} Jul 27 19:42:35.738253 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:35.738011 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f849de3-35a1-40b9-a7b1-817e1ea3a0f0","Type":"ContainerStarted","Data":"1065d61b34742958769abe7ccd67731fae6c15315852d5dda531355e72b94896"} Jul 27 19:42:35.771125 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:35.771064 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.7559428329999998 podStartE2EDuration="2.771044081s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.101248ms" isStatefulPod=false podCreationTimestamp="2026-07-27 19:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:42:35.768462483 +0000 UTC m=+160.381047086" watchObservedRunningTime="2026-07-27 19:42:35.771044081 +0000 UTC m=+160.383628665" Jul 27 19:42:36.227795 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:36.227761 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:39.233794 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:39.233758 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:39.331548 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:39.331513 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:41.607552 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.607519 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76dcd887fb-h22qn"] Jul 27 19:42:41.610603 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.610586 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.622058 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.622037 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76dcd887fb-h22qn"] Jul 27 19:42:41.693527 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.693499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-oauth-serving-cert\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.693678 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.693588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r99sh\" (UniqueName: \"kubernetes.io/projected/a4328e67-0b2a-4b56-b002-d2518ffffb11-kube-api-access-r99sh\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.693678 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.693625 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-serving-cert\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.693678 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.693661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-config\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.693821 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.693690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-trusted-ca-bundle\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.693821 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.693732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-oauth-config\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.693821 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.693768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-service-ca\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.694361 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.694329 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:41.749971 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.749948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:41.794422 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.794393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-config\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.794575 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.794430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-trusted-ca-bundle\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.794575 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.794453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-oauth-config\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.794575 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.794467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-service-ca\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.794575 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.794485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-oauth-serving-cert\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.795703 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.795317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99sh\" (UniqueName: \"kubernetes.io/projected/a4328e67-0b2a-4b56-b002-d2518ffffb11-kube-api-access-r99sh\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.795703 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.795393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-serving-cert\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.795703 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.795591 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-oauth-serving-cert\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.795703 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.795641 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-config\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.795703 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.795692 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-service-ca\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.796177 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.796149 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-trusted-ca-bundle\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.798125 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.798103 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-oauth-config\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.798326 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.798284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-serving-cert\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.804403 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.804382 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r99sh\" (UniqueName: \"kubernetes.io/projected/a4328e67-0b2a-4b56-b002-d2518ffffb11-kube-api-access-r99sh\") pod \"console-76dcd887fb-h22qn\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:41.920241 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:41.920153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:42.046145 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:42.046108 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76dcd887fb-h22qn"] Jul 27 19:42:42.048871 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:42:42.048841 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4328e67_0b2a_4b56_b002_d2518ffffb11.slice/crio-762832a05f16285916135b332d4cd7918691e2dea4aba04bd5d10a0643e49be1 WatchSource:0}: Error finding container 762832a05f16285916135b332d4cd7918691e2dea4aba04bd5d10a0643e49be1: Status 404 returned error can't find the container with id 762832a05f16285916135b332d4cd7918691e2dea4aba04bd5d10a0643e49be1 Jul 27 19:42:42.765607 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:42.765569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dcd887fb-h22qn" event={"ID":"a4328e67-0b2a-4b56-b002-d2518ffffb11","Type":"ContainerStarted","Data":"01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5"} Jul 27 19:42:42.765999 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:42.765630 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dcd887fb-h22qn" event={"ID":"a4328e67-0b2a-4b56-b002-d2518ffffb11","Type":"ContainerStarted","Data":"762832a05f16285916135b332d4cd7918691e2dea4aba04bd5d10a0643e49be1"} Jul 27 19:42:42.786198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:42.786145 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76dcd887fb-h22qn" podStartSLOduration=1.786128407 podStartE2EDuration="1.786128407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:42:42.784483266 +0000 UTC m=+167.397067864" watchObservedRunningTime="2026-07-27 19:42:42.786128407 +0000 UTC m=+167.398713057" Jul 27 19:42:47.398842 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:47.398718 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:47.497896 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:47.497837 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:51.921342 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:51.921289 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:51.921694 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:51.921364 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:51.926129 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:51.926105 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:52.808827 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:52.808791 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 19:42:52.870237 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:52.870205 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5867c6bcb5-nzb4g"] Jul 27 19:42:53.048316 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:53.048266 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:42:53.148840 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:42:53.148676 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:13.686106 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:13.686068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:13.787470 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:13.787410 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:14.020151 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.020078 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:14.326124 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.326043 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw"] Jul 27 19:43:14.329742 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.329720 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.332537 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.332519 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gm87r\"" Jul 27 19:43:14.333120 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.333104 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 19:43:14.333687 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.333669 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 19:43:14.340175 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.340152 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw"] Jul 27 19:43:14.364600 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.364577 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.364686 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.364610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.364686 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.364636 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw6kl\" (UniqueName: \"kubernetes.io/projected/7723d608-5b26-4200-b8c0-f46ee6768540-kube-api-access-bw6kl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.465514 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.465484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.465649 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.465525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.465649 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.465552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw6kl\" (UniqueName: \"kubernetes.io/projected/7723d608-5b26-4200-b8c0-f46ee6768540-kube-api-access-bw6kl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.465880 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.465860 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.465923 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.465908 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.475565 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.475539 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw6kl\" (UniqueName: \"kubernetes.io/projected/7723d608-5b26-4200-b8c0-f46ee6768540-kube-api-access-bw6kl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.639517 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.639431 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:14.767357 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.767329 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw"] Jul 27 19:43:14.771241 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:43:14.771214 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7723d608_5b26_4200_b8c0_f46ee6768540.slice/crio-cb83c7639268042c5e35e6f04276205fc3f83c37b17cda76fa930d0a14e9df5a WatchSource:0}: Error finding container cb83c7639268042c5e35e6f04276205fc3f83c37b17cda76fa930d0a14e9df5a: Status 404 returned error can't find the container with id cb83c7639268042c5e35e6f04276205fc3f83c37b17cda76fa930d0a14e9df5a Jul 27 19:43:14.888669 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:14.888637 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" event={"ID":"7723d608-5b26-4200-b8c0-f46ee6768540","Type":"ContainerStarted","Data":"cb83c7639268042c5e35e6f04276205fc3f83c37b17cda76fa930d0a14e9df5a"} Jul 27 19:43:17.846231 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:17.846185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:17.889286 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:17.889230 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5867c6bcb5-nzb4g" podUID="61a1b4f0-9be0-4863-acff-dbce02ecfb90" containerName="console" containerID="cri-o://6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759" gracePeriod=15 Jul 27 19:43:18.142550 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.142523 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:43:18.196026 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.195997 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-oauth-serving-cert\") pod \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " Jul 27 19:43:18.196193 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196065 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-oauth-config\") pod \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " Jul 27 19:43:18.196193 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196139 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-config\") pod \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " Jul 27 19:43:18.196193 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196169 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-serving-cert\") pod \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " Jul 27 19:43:18.196372 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196197 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-service-ca\") pod \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " Jul 27 19:43:18.196372 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196222 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9c5w\" (UniqueName: \"kubernetes.io/projected/61a1b4f0-9be0-4863-acff-dbce02ecfb90-kube-api-access-g9c5w\") pod \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " Jul 27 19:43:18.196372 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196256 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-trusted-ca-bundle\") pod \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\" (UID: \"61a1b4f0-9be0-4863-acff-dbce02ecfb90\") " Jul 27 19:43:18.196577 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196421 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "61a1b4f0-9be0-4863-acff-dbce02ecfb90" (UID: "61a1b4f0-9be0-4863-acff-dbce02ecfb90"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:43:18.197112 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196618 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-service-ca" (OuterVolumeSpecName: "service-ca") pod "61a1b4f0-9be0-4863-acff-dbce02ecfb90" (UID: "61a1b4f0-9be0-4863-acff-dbce02ecfb90"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:43:18.197229 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196723 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-config" (OuterVolumeSpecName: "console-config") pod "61a1b4f0-9be0-4863-acff-dbce02ecfb90" (UID: "61a1b4f0-9be0-4863-acff-dbce02ecfb90"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:43:18.197229 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.196829 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "61a1b4f0-9be0-4863-acff-dbce02ecfb90" (UID: "61a1b4f0-9be0-4863-acff-dbce02ecfb90"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:43:18.198644 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.198616 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "61a1b4f0-9be0-4863-acff-dbce02ecfb90" (UID: "61a1b4f0-9be0-4863-acff-dbce02ecfb90"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:43:18.198773 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.198708 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a1b4f0-9be0-4863-acff-dbce02ecfb90-kube-api-access-g9c5w" (OuterVolumeSpecName: "kube-api-access-g9c5w") pod "61a1b4f0-9be0-4863-acff-dbce02ecfb90" (UID: "61a1b4f0-9be0-4863-acff-dbce02ecfb90"). InnerVolumeSpecName "kube-api-access-g9c5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:43:18.199034 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.199011 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "61a1b4f0-9be0-4863-acff-dbce02ecfb90" (UID: "61a1b4f0-9be0-4863-acff-dbce02ecfb90"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:43:18.297935 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.297897 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:18.298118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.297939 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:18.298118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.297955 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:18.298118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.297967 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9c5w\" (UniqueName: \"kubernetes.io/projected/61a1b4f0-9be0-4863-acff-dbce02ecfb90-kube-api-access-g9c5w\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:18.298118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.297979 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-trusted-ca-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:18.298118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.297989 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61a1b4f0-9be0-4863-acff-dbce02ecfb90-oauth-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:18.298118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.298000 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61a1b4f0-9be0-4863-acff-dbce02ecfb90-console-oauth-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:18.906809 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.906770 2573 generic.go:356] "Generic (PLEG): container finished" podID="61a1b4f0-9be0-4863-acff-dbce02ecfb90" containerID="6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759" exitCode=0 Jul 27 19:43:18.907236 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.906815 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5867c6bcb5-nzb4g" Jul 27 19:43:18.907236 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.906850 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5867c6bcb5-nzb4g" event={"ID":"61a1b4f0-9be0-4863-acff-dbce02ecfb90","Type":"ContainerDied","Data":"6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759"} Jul 27 19:43:18.907236 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.906915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5867c6bcb5-nzb4g" event={"ID":"61a1b4f0-9be0-4863-acff-dbce02ecfb90","Type":"ContainerDied","Data":"0cdc5cbf9d32be83e968e1f724dc8549b08f470da8e5696ffd668aaa89260c16"} Jul 27 19:43:18.907236 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.906941 2573 scope.go:117] "RemoveContainer" containerID="6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759" Jul 27 19:43:18.917365 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.917342 2573 scope.go:117] "RemoveContainer" containerID="6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759" Jul 27 19:43:18.917668 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:43:18.917648 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759\": container with ID starting with 6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759 not found: ID does not exist" containerID="6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759" Jul 27 19:43:18.917756 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.917715 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759"} err="failed to get container status \"6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759\": rpc error: code = NotFound desc = could not find container \"6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759\": container with ID starting with 6798593d2f4f4a369598b1584da930949d399c9ba72e22360efe62d9b5347759 not found: ID does not exist" Jul 27 19:43:18.929939 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.929908 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5867c6bcb5-nzb4g"] Jul 27 19:43:18.934118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:18.934090 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5867c6bcb5-nzb4g"] Jul 27 19:43:19.876174 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:19.876137 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a1b4f0-9be0-4863-acff-dbce02ecfb90" path="/var/lib/kubelet/pods/61a1b4f0-9be0-4863-acff-dbce02ecfb90/volumes" Jul 27 19:43:21.922671 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:21.922633 2573 generic.go:356] "Generic (PLEG): container finished" podID="7723d608-5b26-4200-b8c0-f46ee6768540" containerID="b9e0ec9b06dbcb2e9d38ca3250dc8a5a718825521f3ab48de2c58fafd5396d41" exitCode=0 Jul 27 19:43:21.923053 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:21.922718 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" event={"ID":"7723d608-5b26-4200-b8c0-f46ee6768540","Type":"ContainerDied","Data":"b9e0ec9b06dbcb2e9d38ca3250dc8a5a718825521f3ab48de2c58fafd5396d41"} Jul 27 19:43:24.935251 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:24.935216 2573 generic.go:356] "Generic (PLEG): container finished" podID="7723d608-5b26-4200-b8c0-f46ee6768540" containerID="ec7b795013fe6be8d3284d8089a0f5771c64ba017176d61281c1ab68001d1c85" exitCode=0 Jul 27 19:43:24.935648 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:24.935258 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" event={"ID":"7723d608-5b26-4200-b8c0-f46ee6768540","Type":"ContainerDied","Data":"ec7b795013fe6be8d3284d8089a0f5771c64ba017176d61281c1ab68001d1c85"} Jul 27 19:43:32.845147 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:32.845117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:32.947141 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:32.947108 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:32.971132 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:32.971094 2573 generic.go:356] "Generic (PLEG): container finished" podID="7723d608-5b26-4200-b8c0-f46ee6768540" containerID="0fd25a1d25f6b7b13da0c224d70e92e25f92c7afacbf7f2a63cf575cf4f5b0c4" exitCode=0 Jul 27 19:43:32.971263 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:32.971166 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" event={"ID":"7723d608-5b26-4200-b8c0-f46ee6768540","Type":"ContainerDied","Data":"0fd25a1d25f6b7b13da0c224d70e92e25f92c7afacbf7f2a63cf575cf4f5b0c4"} Jul 27 19:43:34.097507 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.097480 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:34.235653 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.235558 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-util\") pod \"7723d608-5b26-4200-b8c0-f46ee6768540\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " Jul 27 19:43:34.235653 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.235604 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw6kl\" (UniqueName: \"kubernetes.io/projected/7723d608-5b26-4200-b8c0-f46ee6768540-kube-api-access-bw6kl\") pod \"7723d608-5b26-4200-b8c0-f46ee6768540\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " Jul 27 19:43:34.235886 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.235681 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-bundle\") pod \"7723d608-5b26-4200-b8c0-f46ee6768540\" (UID: \"7723d608-5b26-4200-b8c0-f46ee6768540\") " Jul 27 19:43:34.236391 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.236361 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-bundle" (OuterVolumeSpecName: "bundle") pod "7723d608-5b26-4200-b8c0-f46ee6768540" (UID: "7723d608-5b26-4200-b8c0-f46ee6768540"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:43:34.237889 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.237867 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7723d608-5b26-4200-b8c0-f46ee6768540-kube-api-access-bw6kl" (OuterVolumeSpecName: "kube-api-access-bw6kl") pod "7723d608-5b26-4200-b8c0-f46ee6768540" (UID: "7723d608-5b26-4200-b8c0-f46ee6768540"). InnerVolumeSpecName "kube-api-access-bw6kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:43:34.240173 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.240149 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-util" (OuterVolumeSpecName: "util") pod "7723d608-5b26-4200-b8c0-f46ee6768540" (UID: "7723d608-5b26-4200-b8c0-f46ee6768540"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:43:34.336468 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.336417 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:34.336468 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.336461 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7723d608-5b26-4200-b8c0-f46ee6768540-util\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:34.336468 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.336472 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw6kl\" (UniqueName: \"kubernetes.io/projected/7723d608-5b26-4200-b8c0-f46ee6768540-kube-api-access-bw6kl\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:34.983225 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.983188 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" event={"ID":"7723d608-5b26-4200-b8c0-f46ee6768540","Type":"ContainerDied","Data":"cb83c7639268042c5e35e6f04276205fc3f83c37b17cda76fa930d0a14e9df5a"} Jul 27 19:43:34.983419 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.983232 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zbtxw" Jul 27 19:43:34.983419 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:34.983240 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb83c7639268042c5e35e6f04276205fc3f83c37b17cda76fa930d0a14e9df5a" Jul 27 19:43:40.419924 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:40.419884 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:41.410413 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:41.410381 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:42.395468 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.395427 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct"] Jul 27 19:43:42.395915 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.395869 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a1b4f0-9be0-4863-acff-dbce02ecfb90" containerName="console" Jul 27 19:43:42.395915 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.395887 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a1b4f0-9be0-4863-acff-dbce02ecfb90" containerName="console" Jul 27 19:43:42.395915 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.395899 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7723d608-5b26-4200-b8c0-f46ee6768540" containerName="extract" Jul 27 19:43:42.395915 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.395906 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7723d608-5b26-4200-b8c0-f46ee6768540" containerName="extract" Jul 27 19:43:42.396114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.395919 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7723d608-5b26-4200-b8c0-f46ee6768540" containerName="util" Jul 27 19:43:42.396114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.395926 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7723d608-5b26-4200-b8c0-f46ee6768540" containerName="util" Jul 27 19:43:42.396114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.395942 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7723d608-5b26-4200-b8c0-f46ee6768540" containerName="pull" Jul 27 19:43:42.396114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.395950 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7723d608-5b26-4200-b8c0-f46ee6768540" containerName="pull" Jul 27 19:43:42.396114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.396045 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a1b4f0-9be0-4863-acff-dbce02ecfb90" containerName="console" Jul 27 19:43:42.396114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.396061 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7723d608-5b26-4200-b8c0-f46ee6768540" containerName="extract" Jul 27 19:43:42.398883 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.398861 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" Jul 27 19:43:42.401584 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.401564 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-h7m7q\"" Jul 27 19:43:42.401668 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.401571 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 27 19:43:42.401864 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.401849 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:43:42.416477 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.416460 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct"] Jul 27 19:43:42.504941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.504913 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8hm\" (UniqueName: \"kubernetes.io/projected/30e21388-a94a-4006-83d4-7fcedccbabc8-kube-api-access-tz8hm\") pod \"cert-manager-operator-controller-manager-55c444bf86-lb9ct\" (UID: \"30e21388-a94a-4006-83d4-7fcedccbabc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" Jul 27 19:43:42.505113 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.504956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30e21388-a94a-4006-83d4-7fcedccbabc8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lb9ct\" (UID: \"30e21388-a94a-4006-83d4-7fcedccbabc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" Jul 27 19:43:42.605819 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.605773 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8hm\" (UniqueName: \"kubernetes.io/projected/30e21388-a94a-4006-83d4-7fcedccbabc8-kube-api-access-tz8hm\") pod \"cert-manager-operator-controller-manager-55c444bf86-lb9ct\" (UID: \"30e21388-a94a-4006-83d4-7fcedccbabc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" Jul 27 19:43:42.605965 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.605843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30e21388-a94a-4006-83d4-7fcedccbabc8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lb9ct\" (UID: \"30e21388-a94a-4006-83d4-7fcedccbabc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" Jul 27 19:43:42.606252 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.606234 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30e21388-a94a-4006-83d4-7fcedccbabc8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lb9ct\" (UID: \"30e21388-a94a-4006-83d4-7fcedccbabc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" Jul 27 19:43:42.614951 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.614917 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8hm\" (UniqueName: \"kubernetes.io/projected/30e21388-a94a-4006-83d4-7fcedccbabc8-kube-api-access-tz8hm\") pod \"cert-manager-operator-controller-manager-55c444bf86-lb9ct\" (UID: \"30e21388-a94a-4006-83d4-7fcedccbabc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" Jul 27 19:43:42.707403 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.707380 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" Jul 27 19:43:42.845082 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:42.845051 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct"] Jul 27 19:43:42.847543 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:43:42.847507 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e21388_a94a_4006_83d4_7fcedccbabc8.slice/crio-777e20008e313774baf86fa895ca156fa88161278f34a5b490ac6ea64ea0cc33 WatchSource:0}: Error finding container 777e20008e313774baf86fa895ca156fa88161278f34a5b490ac6ea64ea0cc33: Status 404 returned error can't find the container with id 777e20008e313774baf86fa895ca156fa88161278f34a5b490ac6ea64ea0cc33 Jul 27 19:43:43.015840 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:43.015757 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" event={"ID":"30e21388-a94a-4006-83d4-7fcedccbabc8","Type":"ContainerStarted","Data":"777e20008e313774baf86fa895ca156fa88161278f34a5b490ac6ea64ea0cc33"} Jul 27 19:43:44.035517 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:44.035487 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:44.136984 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:44.136951 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:45.026214 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:45.026173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" event={"ID":"30e21388-a94a-4006-83d4-7fcedccbabc8","Type":"ContainerStarted","Data":"bde15e798f3680638a7e5d869e9c5b2283ba9ab9bfc85c3617bb6f999834891e"} Jul 27 19:43:45.049146 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:45.049091 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lb9ct" podStartSLOduration=1.444974455 podStartE2EDuration="3.049074037s" totalImagesPullingTime="1.604099582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:43:45.047461825 +0000 UTC m=+229.660046429" watchObservedRunningTime="2026-07-27 19:43:45.049074037 +0000 UTC m=+229.661658597" Jul 27 19:43:45.463218 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:45.463186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:45.594464 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:45.594430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:46.139984 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.139948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:46.243546 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.243518 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:46.284063 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.284032 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb"] Jul 27 19:43:46.286340 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.286325 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.288948 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.288924 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 19:43:46.289086 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.289042 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 19:43:46.290085 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.290066 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gm87r\"" Jul 27 19:43:46.297529 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.297508 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb"] Jul 27 19:43:46.435891 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.435859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.436077 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.435900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tln46\" (UniqueName: \"kubernetes.io/projected/d0809cc1-f8a7-489c-b310-8096c12264c0-kube-api-access-tln46\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.436077 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.435966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.537233 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.537193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.537465 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.537322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.537465 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.537351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tln46\" (UniqueName: \"kubernetes.io/projected/d0809cc1-f8a7-489c-b310-8096c12264c0-kube-api-access-tln46\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.537621 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.537600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.537670 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.537638 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.546752 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.546726 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tln46\" (UniqueName: \"kubernetes.io/projected/d0809cc1-f8a7-489c-b310-8096c12264c0-kube-api-access-tln46\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.595108 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.595067 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:46.728500 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:46.728389 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb"] Jul 27 19:43:46.730926 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:43:46.730897 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0809cc1_f8a7_489c_b310_8096c12264c0.slice/crio-3258a6ef7eea273809f5df0a83cd526bae84c080a23aa214370268ed1f9ef155 WatchSource:0}: Error finding container 3258a6ef7eea273809f5df0a83cd526bae84c080a23aa214370268ed1f9ef155: Status 404 returned error can't find the container with id 3258a6ef7eea273809f5df0a83cd526bae84c080a23aa214370268ed1f9ef155 Jul 27 19:43:47.036046 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.035958 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerID="9e9ccc3c3da529f0a9e8fde5c04f0f362e150df0e1b5cc4c0a35aecb49910b4b" exitCode=0 Jul 27 19:43:47.036184 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.036046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" event={"ID":"d0809cc1-f8a7-489c-b310-8096c12264c0","Type":"ContainerDied","Data":"9e9ccc3c3da529f0a9e8fde5c04f0f362e150df0e1b5cc4c0a35aecb49910b4b"} Jul 27 19:43:47.036184 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.036091 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" event={"ID":"d0809cc1-f8a7-489c-b310-8096c12264c0","Type":"ContainerStarted","Data":"3258a6ef7eea273809f5df0a83cd526bae84c080a23aa214370268ed1f9ef155"} Jul 27 19:43:47.150952 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.150916 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xms4d"] Jul 27 19:43:47.153188 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.153172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:47.156006 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.155975 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 27 19:43:47.156148 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.156032 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 27 19:43:47.156336 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.156320 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-77knx\"" Jul 27 19:43:47.164862 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.164842 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xms4d"] Jul 27 19:43:47.243643 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.243602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98f3cebc-2965-4537-a220-60eed2756b4a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xms4d\" (UID: \"98f3cebc-2965-4537-a220-60eed2756b4a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:47.243827 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.243748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9ng\" (UniqueName: \"kubernetes.io/projected/98f3cebc-2965-4537-a220-60eed2756b4a-kube-api-access-5g9ng\") pod \"cert-manager-webhook-6486ffd997-xms4d\" (UID: \"98f3cebc-2965-4537-a220-60eed2756b4a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:47.344942 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.344850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9ng\" (UniqueName: \"kubernetes.io/projected/98f3cebc-2965-4537-a220-60eed2756b4a-kube-api-access-5g9ng\") pod \"cert-manager-webhook-6486ffd997-xms4d\" (UID: \"98f3cebc-2965-4537-a220-60eed2756b4a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:47.344942 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.344898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98f3cebc-2965-4537-a220-60eed2756b4a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xms4d\" (UID: \"98f3cebc-2965-4537-a220-60eed2756b4a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:47.354995 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.354968 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98f3cebc-2965-4537-a220-60eed2756b4a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xms4d\" (UID: \"98f3cebc-2965-4537-a220-60eed2756b4a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:47.355112 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.355035 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9ng\" (UniqueName: \"kubernetes.io/projected/98f3cebc-2965-4537-a220-60eed2756b4a-kube-api-access-5g9ng\") pod \"cert-manager-webhook-6486ffd997-xms4d\" (UID: \"98f3cebc-2965-4537-a220-60eed2756b4a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:47.473218 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.473183 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:47.609857 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:47.609786 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xms4d"] Jul 27 19:43:47.613039 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:43:47.613012 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98f3cebc_2965_4537_a220_60eed2756b4a.slice/crio-fc409a6ee0d738397c88d862778b399b83738f883f82b264ff1ae70caff80da2 WatchSource:0}: Error finding container fc409a6ee0d738397c88d862778b399b83738f883f82b264ff1ae70caff80da2: Status 404 returned error can't find the container with id fc409a6ee0d738397c88d862778b399b83738f883f82b264ff1ae70caff80da2 Jul 27 19:43:48.042975 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.042941 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" event={"ID":"98f3cebc-2965-4537-a220-60eed2756b4a","Type":"ContainerStarted","Data":"fc409a6ee0d738397c88d862778b399b83738f883f82b264ff1ae70caff80da2"} Jul 27 19:43:48.342505 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.342427 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:48.444781 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.444750 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p"] Jul 27 19:43:48.446818 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.446788 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" Jul 27 19:43:48.450499 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.450465 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sflrz\"" Jul 27 19:43:48.458362 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.458340 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p"] Jul 27 19:43:48.550906 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.550868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:48.557045 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.557007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvn2\" (UniqueName: \"kubernetes.io/projected/f3cbec6a-62b4-47c5-8989-1c91333fd6bc-kube-api-access-dhvn2\") pod \"cert-manager-cainjector-559cdd8c87-pkw4p\" (UID: \"f3cbec6a-62b4-47c5-8989-1c91333fd6bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" Jul 27 19:43:48.557204 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.557093 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3cbec6a-62b4-47c5-8989-1c91333fd6bc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pkw4p\" (UID: \"f3cbec6a-62b4-47c5-8989-1c91333fd6bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" Jul 27 19:43:48.658554 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.658472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvn2\" (UniqueName: \"kubernetes.io/projected/f3cbec6a-62b4-47c5-8989-1c91333fd6bc-kube-api-access-dhvn2\") pod \"cert-manager-cainjector-559cdd8c87-pkw4p\" (UID: \"f3cbec6a-62b4-47c5-8989-1c91333fd6bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" Jul 27 19:43:48.658554 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.658539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3cbec6a-62b4-47c5-8989-1c91333fd6bc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pkw4p\" (UID: \"f3cbec6a-62b4-47c5-8989-1c91333fd6bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" Jul 27 19:43:48.668265 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.668238 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3cbec6a-62b4-47c5-8989-1c91333fd6bc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pkw4p\" (UID: \"f3cbec6a-62b4-47c5-8989-1c91333fd6bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" Jul 27 19:43:48.668421 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.668276 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvn2\" (UniqueName: \"kubernetes.io/projected/f3cbec6a-62b4-47c5-8989-1c91333fd6bc-kube-api-access-dhvn2\") pod \"cert-manager-cainjector-559cdd8c87-pkw4p\" (UID: \"f3cbec6a-62b4-47c5-8989-1c91333fd6bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" Jul 27 19:43:48.757170 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:48.757131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" Jul 27 19:43:49.308706 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:49.308572 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p"] Jul 27 19:43:49.311450 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:43:49.311424 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3cbec6a_62b4_47c5_8989_1c91333fd6bc.slice/crio-61b5be07a1f6c89bcdde348278ab8d177705d08e6aa030f7baa503317ef8682c WatchSource:0}: Error finding container 61b5be07a1f6c89bcdde348278ab8d177705d08e6aa030f7baa503317ef8682c: Status 404 returned error can't find the container with id 61b5be07a1f6c89bcdde348278ab8d177705d08e6aa030f7baa503317ef8682c Jul 27 19:43:50.054840 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:50.054806 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerID="dcd1fea9674734910e41fd37db4d2d1d3b236d4d0bdf423abb58f31da16459ca" exitCode=0 Jul 27 19:43:50.055346 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:50.054893 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" event={"ID":"d0809cc1-f8a7-489c-b310-8096c12264c0","Type":"ContainerDied","Data":"dcd1fea9674734910e41fd37db4d2d1d3b236d4d0bdf423abb58f31da16459ca"} Jul 27 19:43:50.056371 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:50.056337 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" event={"ID":"f3cbec6a-62b4-47c5-8989-1c91333fd6bc","Type":"ContainerStarted","Data":"61b5be07a1f6c89bcdde348278ab8d177705d08e6aa030f7baa503317ef8682c"} Jul 27 19:43:50.832091 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:50.832050 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:50.932778 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:50.932742 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:51.064256 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:51.064221 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerID="53e36558107a4474aefe85db63b3b9a5c72ff828d7767fa9ddf27447f91f6c44" exitCode=0 Jul 27 19:43:51.064703 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:51.064324 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" event={"ID":"d0809cc1-f8a7-489c-b310-8096c12264c0","Type":"ContainerDied","Data":"53e36558107a4474aefe85db63b3b9a5c72ff828d7767fa9ddf27447f91f6c44"} Jul 27 19:43:52.070171 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.070139 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" event={"ID":"f3cbec6a-62b4-47c5-8989-1c91333fd6bc","Type":"ContainerStarted","Data":"b73df2e83f8891a890c18b325eb6047fbb968029e460303299547cf31e58662d"} Jul 27 19:43:52.071744 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.071717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" event={"ID":"98f3cebc-2965-4537-a220-60eed2756b4a","Type":"ContainerStarted","Data":"1d42c85ceb1b71452c72d450aac4325db65a79cc9521b93fe0e9e2ce770987e8"} Jul 27 19:43:52.071883 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.071836 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:52.089945 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.089874 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pkw4p" podStartSLOduration=2.1702821549999998 podStartE2EDuration="4.089852596s" totalImagesPullingTime="1.919570441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:43:52.087967451 +0000 UTC m=+236.700552036" watchObservedRunningTime="2026-07-27 19:43:52.089852596 +0000 UTC m=+236.702437195" Jul 27 19:43:52.108228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.108161 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" podStartSLOduration=1.490465631 podStartE2EDuration="5.108138291s" totalImagesPullingTime="3.61767266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:43:52.106461114 +0000 UTC m=+236.719045699" watchObservedRunningTime="2026-07-27 19:43:52.108138291 +0000 UTC m=+236.720722875" Jul 27 19:43:52.209126 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.209101 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:52.291225 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.291200 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-bundle\") pod \"d0809cc1-f8a7-489c-b310-8096c12264c0\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " Jul 27 19:43:52.291385 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.291260 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tln46\" (UniqueName: \"kubernetes.io/projected/d0809cc1-f8a7-489c-b310-8096c12264c0-kube-api-access-tln46\") pod \"d0809cc1-f8a7-489c-b310-8096c12264c0\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " Jul 27 19:43:52.291385 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.291357 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-util\") pod \"d0809cc1-f8a7-489c-b310-8096c12264c0\" (UID: \"d0809cc1-f8a7-489c-b310-8096c12264c0\") " Jul 27 19:43:52.291708 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.291686 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-bundle" (OuterVolumeSpecName: "bundle") pod "d0809cc1-f8a7-489c-b310-8096c12264c0" (UID: "d0809cc1-f8a7-489c-b310-8096c12264c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:43:52.293544 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.293519 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0809cc1-f8a7-489c-b310-8096c12264c0-kube-api-access-tln46" (OuterVolumeSpecName: "kube-api-access-tln46") pod "d0809cc1-f8a7-489c-b310-8096c12264c0" (UID: "d0809cc1-f8a7-489c-b310-8096c12264c0"). InnerVolumeSpecName "kube-api-access-tln46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:43:52.295902 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.295875 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-util" (OuterVolumeSpecName: "util") pod "d0809cc1-f8a7-489c-b310-8096c12264c0" (UID: "d0809cc1-f8a7-489c-b310-8096c12264c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:43:52.392172 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.392088 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tln46\" (UniqueName: \"kubernetes.io/projected/d0809cc1-f8a7-489c-b310-8096c12264c0-kube-api-access-tln46\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:52.392172 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.392116 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-util\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:52.392172 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:52.392128 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0809cc1-f8a7-489c-b310-8096c12264c0-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:43:53.078311 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:53.078279 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" Jul 27 19:43:53.078673 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:53.078316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv67fb" event={"ID":"d0809cc1-f8a7-489c-b310-8096c12264c0","Type":"ContainerDied","Data":"3258a6ef7eea273809f5df0a83cd526bae84c080a23aa214370268ed1f9ef155"} Jul 27 19:43:53.078673 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:53.078361 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3258a6ef7eea273809f5df0a83cd526bae84c080a23aa214370268ed1f9ef155" Jul 27 19:43:56.563632 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.563601 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:56.771894 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.771859 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wp57d"] Jul 27 19:43:56.772199 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.772185 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerName="pull" Jul 27 19:43:56.772199 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.772199 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerName="pull" Jul 27 19:43:56.772418 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.772210 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerName="util" Jul 27 19:43:56.772418 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.772216 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerName="util" Jul 27 19:43:56.772418 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.772224 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerName="extract" Jul 27 19:43:56.772418 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.772230 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerName="extract" Jul 27 19:43:56.772418 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.772278 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0809cc1-f8a7-489c-b310-8096c12264c0" containerName="extract" Jul 27 19:43:56.775528 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.775511 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wp57d" Jul 27 19:43:56.779190 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.779169 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-q9tdh\"" Jul 27 19:43:56.798341 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.798314 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wp57d"] Jul 27 19:43:56.934391 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.934359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgnck\" (UniqueName: \"kubernetes.io/projected/e11661cd-22fe-4cbf-ae93-a96a6d20a15b-kube-api-access-jgnck\") pod \"cert-manager-55b4d954d5-wp57d\" (UID: \"e11661cd-22fe-4cbf-ae93-a96a6d20a15b\") " pod="cert-manager/cert-manager-55b4d954d5-wp57d" Jul 27 19:43:56.934560 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:56.934444 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e11661cd-22fe-4cbf-ae93-a96a6d20a15b-bound-sa-token\") pod \"cert-manager-55b4d954d5-wp57d\" (UID: \"e11661cd-22fe-4cbf-ae93-a96a6d20a15b\") " pod="cert-manager/cert-manager-55b4d954d5-wp57d" Jul 27 19:43:57.035023 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:57.034982 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgnck\" (UniqueName: \"kubernetes.io/projected/e11661cd-22fe-4cbf-ae93-a96a6d20a15b-kube-api-access-jgnck\") pod \"cert-manager-55b4d954d5-wp57d\" (UID: \"e11661cd-22fe-4cbf-ae93-a96a6d20a15b\") " pod="cert-manager/cert-manager-55b4d954d5-wp57d" Jul 27 19:43:57.035023 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:57.035032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e11661cd-22fe-4cbf-ae93-a96a6d20a15b-bound-sa-token\") pod \"cert-manager-55b4d954d5-wp57d\" (UID: \"e11661cd-22fe-4cbf-ae93-a96a6d20a15b\") " pod="cert-manager/cert-manager-55b4d954d5-wp57d" Jul 27 19:43:57.043642 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:57.043619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e11661cd-22fe-4cbf-ae93-a96a6d20a15b-bound-sa-token\") pod \"cert-manager-55b4d954d5-wp57d\" (UID: \"e11661cd-22fe-4cbf-ae93-a96a6d20a15b\") " pod="cert-manager/cert-manager-55b4d954d5-wp57d" Jul 27 19:43:57.043793 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:57.043770 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgnck\" (UniqueName: \"kubernetes.io/projected/e11661cd-22fe-4cbf-ae93-a96a6d20a15b-kube-api-access-jgnck\") pod \"cert-manager-55b4d954d5-wp57d\" (UID: \"e11661cd-22fe-4cbf-ae93-a96a6d20a15b\") " pod="cert-manager/cert-manager-55b4d954d5-wp57d" Jul 27 19:43:57.084761 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:57.084739 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wp57d" Jul 27 19:43:57.218034 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:57.218007 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wp57d"] Jul 27 19:43:57.220500 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:43:57.220461 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode11661cd_22fe_4cbf_ae93_a96a6d20a15b.slice/crio-ce0068278c940a65d2767a2dc6ea94eb5e09924c58da6ab477646e9f5e37fafb WatchSource:0}: Error finding container ce0068278c940a65d2767a2dc6ea94eb5e09924c58da6ab477646e9f5e37fafb: Status 404 returned error can't find the container with id ce0068278c940a65d2767a2dc6ea94eb5e09924c58da6ab477646e9f5e37fafb Jul 27 19:43:57.248062 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:57.248043 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:57.348812 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:57.348779 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:43:58.080663 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:58.080637 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xms4d" Jul 27 19:43:58.102524 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:58.102498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wp57d" event={"ID":"e11661cd-22fe-4cbf-ae93-a96a6d20a15b","Type":"ContainerStarted","Data":"825ecf88a5f69b42afee3110a5c4e543b4746c7ad45e0723838e04e5c68321d1"} Jul 27 19:43:58.102678 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:58.102536 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wp57d" event={"ID":"e11661cd-22fe-4cbf-ae93-a96a6d20a15b","Type":"ContainerStarted","Data":"ce0068278c940a65d2767a2dc6ea94eb5e09924c58da6ab477646e9f5e37fafb"} Jul 27 19:43:58.124229 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:58.124180 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-wp57d" podStartSLOduration=2.124164957 podStartE2EDuration="2.124164957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:43:58.122399046 +0000 UTC m=+242.734983629" watchObservedRunningTime="2026-07-27 19:43:58.124164957 +0000 UTC m=+242.736749539" Jul 27 19:43:59.261820 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.261785 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb"] Jul 27 19:43:59.265171 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.265152 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" Jul 27 19:43:59.267587 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.267565 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 27 19:43:59.268919 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.268889 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:43:59.269011 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.268930 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-8qm78\"" Jul 27 19:43:59.280641 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.280621 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb"] Jul 27 19:43:59.455291 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.455257 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3436a5-575a-424b-8c7a-2c1418e8ea27-tmp\") pod \"openshift-lws-operator-fd8ccff4c-h8bkb\" (UID: \"9c3436a5-575a-424b-8c7a-2c1418e8ea27\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" Jul 27 19:43:59.455477 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.455394 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5cqm\" (UniqueName: \"kubernetes.io/projected/9c3436a5-575a-424b-8c7a-2c1418e8ea27-kube-api-access-d5cqm\") pod \"openshift-lws-operator-fd8ccff4c-h8bkb\" (UID: \"9c3436a5-575a-424b-8c7a-2c1418e8ea27\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" Jul 27 19:43:59.556744 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.556663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5cqm\" (UniqueName: \"kubernetes.io/projected/9c3436a5-575a-424b-8c7a-2c1418e8ea27-kube-api-access-d5cqm\") pod \"openshift-lws-operator-fd8ccff4c-h8bkb\" (UID: \"9c3436a5-575a-424b-8c7a-2c1418e8ea27\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" Jul 27 19:43:59.556744 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.556706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3436a5-575a-424b-8c7a-2c1418e8ea27-tmp\") pod \"openshift-lws-operator-fd8ccff4c-h8bkb\" (UID: \"9c3436a5-575a-424b-8c7a-2c1418e8ea27\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" Jul 27 19:43:59.557022 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.557005 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3436a5-575a-424b-8c7a-2c1418e8ea27-tmp\") pod \"openshift-lws-operator-fd8ccff4c-h8bkb\" (UID: \"9c3436a5-575a-424b-8c7a-2c1418e8ea27\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" Jul 27 19:43:59.566462 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.566438 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5cqm\" (UniqueName: \"kubernetes.io/projected/9c3436a5-575a-424b-8c7a-2c1418e8ea27-kube-api-access-d5cqm\") pod \"openshift-lws-operator-fd8ccff4c-h8bkb\" (UID: \"9c3436a5-575a-424b-8c7a-2c1418e8ea27\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" Jul 27 19:43:59.574248 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.574226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" Jul 27 19:43:59.702399 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:43:59.702348 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb"] Jul 27 19:43:59.704986 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:43:59.704965 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c3436a5_575a_424b_8c7a_2c1418e8ea27.slice/crio-3c89af8b9d27b33a33998e53cac24030b1e4ef45216596fef98b7af81ccb7cdb WatchSource:0}: Error finding container 3c89af8b9d27b33a33998e53cac24030b1e4ef45216596fef98b7af81ccb7cdb: Status 404 returned error can't find the container with id 3c89af8b9d27b33a33998e53cac24030b1e4ef45216596fef98b7af81ccb7cdb Jul 27 19:44:00.112746 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:00.112712 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" event={"ID":"9c3436a5-575a-424b-8c7a-2c1418e8ea27","Type":"ContainerStarted","Data":"3c89af8b9d27b33a33998e53cac24030b1e4ef45216596fef98b7af81ccb7cdb"} Jul 27 19:44:02.123631 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:02.123594 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" event={"ID":"9c3436a5-575a-424b-8c7a-2c1418e8ea27","Type":"ContainerStarted","Data":"5f5b224cf4fe35b0caeed3a9bf809da4fd81c768f1a47593a1162ba2e0a1d464"} Jul 27 19:44:02.138850 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:02.138790 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h8bkb" podStartSLOduration=1.567932726 podStartE2EDuration="3.138771151s" totalImagesPullingTime="1.570838425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:02.138545951 +0000 UTC m=+246.751130531" watchObservedRunningTime="2026-07-27 19:44:02.138771151 +0000 UTC m=+246.751355734" Jul 27 19:44:02.816525 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:02.816491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:02.918342 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:02.918281 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:04.748920 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:04.748890 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:07.142579 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.142533 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499"] Jul 27 19:44:07.165093 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.165060 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499"] Jul 27 19:44:07.165238 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.165165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.167982 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.167955 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 19:44:07.167982 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.167978 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 19:44:07.168221 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.168208 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gm87r\"" Jul 27 19:44:07.214589 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.214556 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.214714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.214630 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2mmf\" (UniqueName: \"kubernetes.io/projected/6356e0d6-d0a3-46db-a7b7-680930a83824-kube-api-access-l2mmf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.214714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.214659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.315919 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.315879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2mmf\" (UniqueName: \"kubernetes.io/projected/6356e0d6-d0a3-46db-a7b7-680930a83824-kube-api-access-l2mmf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.315919 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.315923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.316127 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.315969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.316436 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.316416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.316436 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.316428 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.325388 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.325364 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2mmf\" (UniqueName: \"kubernetes.io/projected/6356e0d6-d0a3-46db-a7b7-680930a83824-kube-api-access-l2mmf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.474715 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.474683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:07.627481 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:07.627454 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499"] Jul 27 19:44:07.629652 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:44:07.629625 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6356e0d6_d0a3_46db_a7b7_680930a83824.slice/crio-d8c08d9a079b5fd122e7b0e839356e12b7b44efc1d2bb57953168f9042bc6e19 WatchSource:0}: Error finding container d8c08d9a079b5fd122e7b0e839356e12b7b44efc1d2bb57953168f9042bc6e19: Status 404 returned error can't find the container with id d8c08d9a079b5fd122e7b0e839356e12b7b44efc1d2bb57953168f9042bc6e19 Jul 27 19:44:08.151903 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:08.151824 2573 generic.go:356] "Generic (PLEG): container finished" podID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerID="13b7a6ecf78cd528079272ce51962843748069d32390cd77457c3637f9d23ba5" exitCode=0 Jul 27 19:44:08.151903 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:08.151892 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" event={"ID":"6356e0d6-d0a3-46db-a7b7-680930a83824","Type":"ContainerDied","Data":"13b7a6ecf78cd528079272ce51962843748069d32390cd77457c3637f9d23ba5"} Jul 27 19:44:08.152332 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:08.151931 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" event={"ID":"6356e0d6-d0a3-46db-a7b7-680930a83824","Type":"ContainerStarted","Data":"d8c08d9a079b5fd122e7b0e839356e12b7b44efc1d2bb57953168f9042bc6e19"} Jul 27 19:44:09.158484 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:09.158452 2573 generic.go:356] "Generic (PLEG): container finished" podID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerID="7673819c69a2d0f1206dfee087687efc63f89d1a9c3526e0fe9dc7cc71950a62" exitCode=0 Jul 27 19:44:09.158993 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:09.158509 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" event={"ID":"6356e0d6-d0a3-46db-a7b7-680930a83824","Type":"ContainerDied","Data":"7673819c69a2d0f1206dfee087687efc63f89d1a9c3526e0fe9dc7cc71950a62"} Jul 27 19:44:10.164522 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:10.164488 2573 generic.go:356] "Generic (PLEG): container finished" podID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerID="ea8ee6348bb8a5bb12bdee20d181f8eca2630a75ea6ed3ee81a3fa7c8db56775" exitCode=0 Jul 27 19:44:10.164899 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:10.164583 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" event={"ID":"6356e0d6-d0a3-46db-a7b7-680930a83824","Type":"ContainerDied","Data":"ea8ee6348bb8a5bb12bdee20d181f8eca2630a75ea6ed3ee81a3fa7c8db56775"} Jul 27 19:44:11.295386 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.295360 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:11.349776 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.349743 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-util\") pod \"6356e0d6-d0a3-46db-a7b7-680930a83824\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " Jul 27 19:44:11.349937 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.349825 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2mmf\" (UniqueName: \"kubernetes.io/projected/6356e0d6-d0a3-46db-a7b7-680930a83824-kube-api-access-l2mmf\") pod \"6356e0d6-d0a3-46db-a7b7-680930a83824\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " Jul 27 19:44:11.349937 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.349884 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-bundle\") pod \"6356e0d6-d0a3-46db-a7b7-680930a83824\" (UID: \"6356e0d6-d0a3-46db-a7b7-680930a83824\") " Jul 27 19:44:11.350586 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.350556 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-bundle" (OuterVolumeSpecName: "bundle") pod "6356e0d6-d0a3-46db-a7b7-680930a83824" (UID: "6356e0d6-d0a3-46db-a7b7-680930a83824"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:44:11.352014 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.351988 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6356e0d6-d0a3-46db-a7b7-680930a83824-kube-api-access-l2mmf" (OuterVolumeSpecName: "kube-api-access-l2mmf") pod "6356e0d6-d0a3-46db-a7b7-680930a83824" (UID: "6356e0d6-d0a3-46db-a7b7-680930a83824"). InnerVolumeSpecName "kube-api-access-l2mmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:44:11.355175 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.355154 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-util" (OuterVolumeSpecName: "util") pod "6356e0d6-d0a3-46db-a7b7-680930a83824" (UID: "6356e0d6-d0a3-46db-a7b7-680930a83824"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:44:11.450824 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.450795 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:44:11.450824 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.450824 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6356e0d6-d0a3-46db-a7b7-680930a83824-util\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:44:11.451001 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:11.450835 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2mmf\" (UniqueName: \"kubernetes.io/projected/6356e0d6-d0a3-46db-a7b7-680930a83824-kube-api-access-l2mmf\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:44:12.175827 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:12.175743 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" Jul 27 19:44:12.175827 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:12.175761 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4q499" event={"ID":"6356e0d6-d0a3-46db-a7b7-680930a83824","Type":"ContainerDied","Data":"d8c08d9a079b5fd122e7b0e839356e12b7b44efc1d2bb57953168f9042bc6e19"} Jul 27 19:44:12.175827 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:12.175805 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8c08d9a079b5fd122e7b0e839356e12b7b44efc1d2bb57953168f9042bc6e19" Jul 27 19:44:14.751619 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:14.751584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:14.815414 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:14.815384 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:15.404253 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:15.404217 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:15.468887 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:15.468856 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:19.060807 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:19.060728 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:19.166858 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:19.166822 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:19.877746 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:19.877711 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:20.105212 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.105181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:20.209965 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.209926 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:20.286812 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.286779 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz"] Jul 27 19:44:20.287311 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.287275 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerName="extract" Jul 27 19:44:20.287363 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.287332 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerName="extract" Jul 27 19:44:20.287363 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.287354 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerName="util" Jul 27 19:44:20.287427 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.287364 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerName="util" Jul 27 19:44:20.287427 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.287373 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerName="pull" Jul 27 19:44:20.287427 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.287381 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerName="pull" Jul 27 19:44:20.287516 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.287470 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6356e0d6-d0a3-46db-a7b7-680930a83824" containerName="extract" Jul 27 19:44:20.291783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.291764 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.307457 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.307435 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6clqf\"" Jul 27 19:44:20.307588 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.307570 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 19:44:20.307848 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.307834 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 19:44:20.308322 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.308286 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 19:44:20.308813 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.308800 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 19:44:20.326729 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.326705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02970dfb-7f1c-4344-814c-8efe0ff90515-tmp\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.326835 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.326758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02970dfb-7f1c-4344-814c-8efe0ff90515-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.326835 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.326799 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02970dfb-7f1c-4344-814c-8efe0ff90515-manifests\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.326835 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.326817 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02970dfb-7f1c-4344-814c-8efe0ff90515-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.326948 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.326843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8md\" (UniqueName: \"kubernetes.io/projected/02970dfb-7f1c-4344-814c-8efe0ff90515-kube-api-access-ps8md\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.341117 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.341086 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz"] Jul 27 19:44:20.427683 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.427650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02970dfb-7f1c-4344-814c-8efe0ff90515-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.427683 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.427690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02970dfb-7f1c-4344-814c-8efe0ff90515-manifests\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.427931 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.427707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02970dfb-7f1c-4344-814c-8efe0ff90515-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.427931 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.427731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8md\" (UniqueName: \"kubernetes.io/projected/02970dfb-7f1c-4344-814c-8efe0ff90515-kube-api-access-ps8md\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.427931 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.427763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02970dfb-7f1c-4344-814c-8efe0ff90515-tmp\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.428151 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.428129 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02970dfb-7f1c-4344-814c-8efe0ff90515-manifests\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.430183 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.430152 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02970dfb-7f1c-4344-814c-8efe0ff90515-tmp\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.430430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.430413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02970dfb-7f1c-4344-814c-8efe0ff90515-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.430473 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.430438 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02970dfb-7f1c-4344-814c-8efe0ff90515-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.439554 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.439530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8md\" (UniqueName: \"kubernetes.io/projected/02970dfb-7f1c-4344-814c-8efe0ff90515-kube-api-access-ps8md\") pod \"opendatahub-operator-controller-manager-7df5588f67-zxvlz\" (UID: \"02970dfb-7f1c-4344-814c-8efe0ff90515\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.480198 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.480136 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:20.580136 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.580105 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:20.602313 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.602262 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:20.694875 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.694841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:20.757521 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.755140 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz"] Jul 27 19:44:20.798854 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.798821 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:20.910120 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:20.910089 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:21.012670 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:21.012591 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:21.219334 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:21.219268 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" event={"ID":"02970dfb-7f1c-4344-814c-8efe0ff90515","Type":"ContainerStarted","Data":"14fe8cf0c586240e32bd3ed3be88bb5604bfb58578f24b2a2a96763472ae3596"} Jul 27 19:44:24.236818 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:24.236786 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" event={"ID":"02970dfb-7f1c-4344-814c-8efe0ff90515","Type":"ContainerStarted","Data":"523ad3549c8071bfbb566b4673319e19b82323133155d80c9f76fc833e4afc31"} Jul 27 19:44:25.121114 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:25.121036 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:25.224445 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:25.224406 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:25.242445 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:25.242415 2573 generic.go:356] "Generic (PLEG): container finished" podID="02970dfb-7f1c-4344-814c-8efe0ff90515" containerID="523ad3549c8071bfbb566b4673319e19b82323133155d80c9f76fc833e4afc31" exitCode=0 Jul 27 19:44:25.242800 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:25.242513 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" event={"ID":"02970dfb-7f1c-4344-814c-8efe0ff90515","Type":"ContainerDied","Data":"523ad3549c8071bfbb566b4673319e19b82323133155d80c9f76fc833e4afc31"} Jul 27 19:44:26.248636 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:26.248599 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" event={"ID":"02970dfb-7f1c-4344-814c-8efe0ff90515","Type":"ContainerStarted","Data":"e1656649912dae546b1558c58780f1bb8d50261f8ac1354eeb28dc071cdd48e7"} Jul 27 19:44:26.249124 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:26.248726 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:26.275022 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:26.274973 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" podStartSLOduration=2.102473899 podStartE2EDuration="6.274957952s" totalImagesPullingTime="3.105630993s" totalInitContainerRuntime="1.06685306s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:26.272225798 +0000 UTC m=+270.884810401" watchObservedRunningTime="2026-07-27 19:44:26.274957952 +0000 UTC m=+270.887542535" Jul 27 19:44:37.255130 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:37.255101 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-zxvlz" Jul 27 19:44:38.918112 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:38.918078 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:39.070237 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:39.070177 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:39.424840 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:39.424809 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:39.689137 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:39.689100 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:39.941121 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:39.941034 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:40.044621 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:40.044580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:40.142018 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:40.141950 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:40.246197 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:40.246119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:40.782988 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:40.782956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:40.997393 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:40.997364 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:41.078714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:41.078640 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:42.184307 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:42.184274 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:47.398963 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.398930 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk"] Jul 27 19:44:47.401225 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.401208 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.404181 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.404164 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 19:44:47.404405 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.404384 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 27 19:44:47.404519 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.404388 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 27 19:44:47.404519 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.404428 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-889c2\"" Jul 27 19:44:47.404519 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.404393 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 19:44:47.404681 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.404637 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 19:44:47.404738 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.404722 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 27 19:44:47.415425 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.415405 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk"] Jul 27 19:44:47.503394 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.503360 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:47.569314 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.569263 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/642a4ebe-1b8e-4003-984e-74a01f4827d8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.569503 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.569382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/642a4ebe-1b8e-4003-984e-74a01f4827d8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.569503 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.569406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czkz\" (UniqueName: \"kubernetes.io/projected/642a4ebe-1b8e-4003-984e-74a01f4827d8-kube-api-access-4czkz\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.569503 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.569486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.569650 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.569526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.569650 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.569580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.569650 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.569600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.619195 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.619167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:47.670417 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.670321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/642a4ebe-1b8e-4003-984e-74a01f4827d8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.670417 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.670402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/642a4ebe-1b8e-4003-984e-74a01f4827d8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.670714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.670434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czkz\" (UniqueName: \"kubernetes.io/projected/642a4ebe-1b8e-4003-984e-74a01f4827d8-kube-api-access-4czkz\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.670714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.670475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.670714 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.670513 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.670999 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.670950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.671092 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.671041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.671895 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.671836 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.673457 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.673413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.674156 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.674138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/642a4ebe-1b8e-4003-984e-74a01f4827d8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.675626 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.674399 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.675626 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.674692 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/642a4ebe-1b8e-4003-984e-74a01f4827d8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.680282 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.680235 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czkz\" (UniqueName: \"kubernetes.io/projected/642a4ebe-1b8e-4003-984e-74a01f4827d8-kube-api-access-4czkz\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.684710 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.684664 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642a4ebe-1b8e-4003-984e-74a01f4827d8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-snmmk\" (UID: \"642a4ebe-1b8e-4003-984e-74a01f4827d8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.710437 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.710402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:47.863735 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.863706 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk"] Jul 27 19:44:47.865392 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:44:47.865355 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642a4ebe_1b8e_4003_984e_74a01f4827d8.slice/crio-b566a062e85ba77803beedaedfaf1e2ad9960fd0d2185c57de8a381ea94a859e WatchSource:0}: Error finding container b566a062e85ba77803beedaedfaf1e2ad9960fd0d2185c57de8a381ea94a859e: Status 404 returned error can't find the container with id b566a062e85ba77803beedaedfaf1e2ad9960fd0d2185c57de8a381ea94a859e Jul 27 19:44:47.968525 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.968495 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm"] Jul 27 19:44:47.971577 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.971558 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:47.974541 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.974515 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 27 19:44:47.975245 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.975229 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 27 19:44:47.975344 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.975320 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 27 19:44:47.975592 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.975576 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wl6p8\"" Jul 27 19:44:47.984579 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:47.984558 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm"] Jul 27 19:44:48.074564 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.074533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-manager-config\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.074748 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.074609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtdqk\" (UniqueName: \"kubernetes.io/projected/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-kube-api-access-jtdqk\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.074748 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.074639 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-cert\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.074748 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.074672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-metrics-cert\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.175605 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.175566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-metrics-cert\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.175783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.175666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-manager-config\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.175783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.175732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtdqk\" (UniqueName: \"kubernetes.io/projected/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-kube-api-access-jtdqk\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.175856 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.175789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-cert\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.176506 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.176484 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-manager-config\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.178413 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.178392 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-metrics-cert\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.179251 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.179231 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-cert\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.187746 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.187725 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtdqk\" (UniqueName: \"kubernetes.io/projected/e0ab9eff-a9f1-444a-bec3-0efe9c991f9a-kube-api-access-jtdqk\") pod \"lws-controller-manager-8689f9464-r7dpm\" (UID: \"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a\") " pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.281097 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.280996 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:48.372120 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.372038 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" event={"ID":"642a4ebe-1b8e-4003-984e-74a01f4827d8","Type":"ContainerStarted","Data":"b566a062e85ba77803beedaedfaf1e2ad9960fd0d2185c57de8a381ea94a859e"} Jul 27 19:44:48.390085 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.390052 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:48.430548 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.430523 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm"] Jul 27 19:44:48.432555 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:44:48.432508 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ab9eff_a9f1_444a_bec3_0efe9c991f9a.slice/crio-718d17e4c265b736152478d2e2aeda25648c58bf380f5563efd8604234ad2ee6 WatchSource:0}: Error finding container 718d17e4c265b736152478d2e2aeda25648c58bf380f5563efd8604234ad2ee6: Status 404 returned error can't find the container with id 718d17e4c265b736152478d2e2aeda25648c58bf380f5563efd8604234ad2ee6 Jul 27 19:44:48.712589 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:48.712544 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:49.379197 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:49.379148 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" event={"ID":"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a","Type":"ContainerStarted","Data":"718d17e4c265b736152478d2e2aeda25648c58bf380f5563efd8604234ad2ee6"} Jul 27 19:44:49.775665 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:49.775613 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:50.569089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:50.569045 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:44:50.569180 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:50.569162 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 19:44:50.687640 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:50.687550 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:50.822560 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:50.822454 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:51.093766 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.093691 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:51.300354 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.300323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:51.390438 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.390352 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" event={"ID":"e0ab9eff-a9f1-444a-bec3-0efe9c991f9a","Type":"ContainerStarted","Data":"cea4ec9caec857dc0bc800649753a4ea525141aa9ba742810c69cfbb3d4fee75"} Jul 27 19:44:51.390614 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.390473 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:44:51.392116 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.392092 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" event={"ID":"642a4ebe-1b8e-4003-984e-74a01f4827d8","Type":"ContainerStarted","Data":"b24eefe7fd2933cdac45642dd0e4d5b276061cd643c1a186f1dfb6a0734050d0"} Jul 27 19:44:51.392240 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.392219 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:51.393912 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.393886 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-snmmk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:44:51.394030 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.393933 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" podUID="642a4ebe-1b8e-4003-984e-74a01f4827d8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:44:51.456500 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.456466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:44:51.474077 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.474024 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" podStartSLOduration=2.338720053 podStartE2EDuration="4.474005932s" totalImagesPullingTime="2.135285879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:51.472738607 +0000 UTC m=+296.085323191" watchObservedRunningTime="2026-07-27 19:44:51.474005932 +0000 UTC m=+296.086590514" Jul 27 19:44:51.526209 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:51.526147 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" podStartSLOduration=1.824789361 podStartE2EDuration="4.526127689s" totalImagesPullingTime="2.701338328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:51.525934332 +0000 UTC m=+296.138518916" watchObservedRunningTime="2026-07-27 19:44:51.526127689 +0000 UTC m=+296.138712272" Jul 27 19:44:52.398262 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:52.398236 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-snmmk" Jul 27 19:44:55.765214 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:44:55.765185 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 19:45:02.399558 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:02.399529 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8689f9464-r7dpm" Jul 27 19:45:40.322373 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:40.322339 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:45:40.421425 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:40.421380 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:45:40.505738 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:40.505699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:45:41.218109 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:41.218083 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:45:41.445611 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:41.445579 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:45:55.834564 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.834495 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw"] Jul 27 19:45:55.837067 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.837051 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:55.839946 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.839925 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 19:45:55.839946 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.839939 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 19:45:55.840799 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.840785 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9przh\"" Jul 27 19:45:55.849564 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.849547 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw"] Jul 27 19:45:55.861636 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.861607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:55.861746 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.861644 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:55.861746 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.861680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8jf\" (UniqueName: \"kubernetes.io/projected/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-kube-api-access-bl8jf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:55.962447 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.962399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:55.962447 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.962454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:55.962703 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.962482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8jf\" (UniqueName: \"kubernetes.io/projected/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-kube-api-access-bl8jf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:55.962779 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.962762 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:55.962888 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.962869 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:55.972672 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:55.972646 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8jf\" (UniqueName: \"kubernetes.io/projected/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-kube-api-access-bl8jf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:56.145346 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.145230 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:56.277219 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.277196 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw"] Jul 27 19:45:56.279898 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:45:56.279872 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b7b3cc_97c2_4ebc_8b94_a462a3daf3b9.slice/crio-e0672da0333959ed5a533acbb522b4c8425da2f6da8b7b36e7937f51f33ed964 WatchSource:0}: Error finding container e0672da0333959ed5a533acbb522b4c8425da2f6da8b7b36e7937f51f33ed964: Status 404 returned error can't find the container with id e0672da0333959ed5a533acbb522b4c8425da2f6da8b7b36e7937f51f33ed964 Jul 27 19:45:56.431769 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.431736 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s"] Jul 27 19:45:56.434566 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.434549 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.445183 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.445148 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s"] Jul 27 19:45:56.467508 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.467477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.467508 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.467509 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7grd\" (UniqueName: \"kubernetes.io/projected/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-kube-api-access-x7grd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.467694 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.467586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.568445 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.568415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.568445 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.568449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7grd\" (UniqueName: \"kubernetes.io/projected/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-kube-api-access-x7grd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.568649 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.568499 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.568808 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.568790 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.568869 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.568850 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.578050 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.578019 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7grd\" (UniqueName: \"kubernetes.io/projected/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-kube-api-access-x7grd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.700765 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.700674 2573 generic.go:356] "Generic (PLEG): container finished" podID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerID="e6abb1bd0c999cbc6e8a89f4e7d55ed48e7209bcb741644414c159e3e361eced" exitCode=0 Jul 27 19:45:56.700916 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.700764 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" event={"ID":"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9","Type":"ContainerDied","Data":"e6abb1bd0c999cbc6e8a89f4e7d55ed48e7209bcb741644414c159e3e361eced"} Jul 27 19:45:56.700916 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.700812 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" event={"ID":"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9","Type":"ContainerStarted","Data":"e0672da0333959ed5a533acbb522b4c8425da2f6da8b7b36e7937f51f33ed964"} Jul 27 19:45:56.701195 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.701181 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 19:45:56.755570 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.755538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:45:56.889962 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:56.889936 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s"] Jul 27 19:45:56.892089 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:45:56.892059 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd341abe1_ac9b_4c10_8171_9dcdf9be1ecf.slice/crio-ae15b75be311ade4dc553e2e4f0c3bbc30b07301b125b976d8cb949bc04ed9d3 WatchSource:0}: Error finding container ae15b75be311ade4dc553e2e4f0c3bbc30b07301b125b976d8cb949bc04ed9d3: Status 404 returned error can't find the container with id ae15b75be311ade4dc553e2e4f0c3bbc30b07301b125b976d8cb949bc04ed9d3 Jul 27 19:45:57.056277 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.056242 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh"] Jul 27 19:45:57.059546 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.059530 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.069742 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.069718 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh"] Jul 27 19:45:57.073627 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.073604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.073735 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.073663 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.073735 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.073712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn57f\" (UniqueName: \"kubernetes.io/projected/13ec8e0e-df8f-418e-9c91-691c7f22082e-kube-api-access-rn57f\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.174446 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.174405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.174636 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.174487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.174636 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.174535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn57f\" (UniqueName: \"kubernetes.io/projected/13ec8e0e-df8f-418e-9c91-691c7f22082e-kube-api-access-rn57f\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.174782 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.174763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.174852 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.174807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.185950 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.185926 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn57f\" (UniqueName: \"kubernetes.io/projected/13ec8e0e-df8f-418e-9c91-691c7f22082e-kube-api-access-rn57f\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.369407 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.369388 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:45:57.431999 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.431971 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm"] Jul 27 19:45:57.435812 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.435792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.449606 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.449573 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm"] Jul 27 19:45:57.477379 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.477355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.477503 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.477392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsl8l\" (UniqueName: \"kubernetes.io/projected/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-kube-api-access-nsl8l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.477503 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.477447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.520170 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.520142 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh"] Jul 27 19:45:57.521849 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:45:57.521823 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ec8e0e_df8f_418e_9c91_691c7f22082e.slice/crio-ade7450480ddb129018832e0ca2e78a663782657d5b0f34f0bd0ad16c3849c40 WatchSource:0}: Error finding container ade7450480ddb129018832e0ca2e78a663782657d5b0f34f0bd0ad16c3849c40: Status 404 returned error can't find the container with id ade7450480ddb129018832e0ca2e78a663782657d5b0f34f0bd0ad16c3849c40 Jul 27 19:45:57.578384 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.578354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.578476 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.578399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsl8l\" (UniqueName: \"kubernetes.io/projected/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-kube-api-access-nsl8l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.578522 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.578471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.578699 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.578682 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.578783 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.578767 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.587454 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.587426 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsl8l\" (UniqueName: \"kubernetes.io/projected/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-kube-api-access-nsl8l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.707730 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.707686 2573 generic.go:356] "Generic (PLEG): container finished" podID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerID="a86cb2d805acf9cd1b2c69d66c1b356121eb89c9ed402446ddb5a58188fb9f05" exitCode=0 Jul 27 19:45:57.707881 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.707777 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" event={"ID":"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9","Type":"ContainerDied","Data":"a86cb2d805acf9cd1b2c69d66c1b356121eb89c9ed402446ddb5a58188fb9f05"} Jul 27 19:45:57.709409 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.709215 2573 generic.go:356] "Generic (PLEG): container finished" podID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerID="95862761bec2e358c97b11f1cb57f68986ce8bb74b5784873c77701d711d580f" exitCode=0 Jul 27 19:45:57.709409 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.709307 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" event={"ID":"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf","Type":"ContainerDied","Data":"95862761bec2e358c97b11f1cb57f68986ce8bb74b5784873c77701d711d580f"} Jul 27 19:45:57.709409 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.709355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" event={"ID":"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf","Type":"ContainerStarted","Data":"ae15b75be311ade4dc553e2e4f0c3bbc30b07301b125b976d8cb949bc04ed9d3"} Jul 27 19:45:57.711055 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.711035 2573 generic.go:356] "Generic (PLEG): container finished" podID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerID="9c869b32f46fa7831f92c4fbfb4dc9cb76a4016363cc10bea0645e285eae1b2e" exitCode=0 Jul 27 19:45:57.711152 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.711062 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" event={"ID":"13ec8e0e-df8f-418e-9c91-691c7f22082e","Type":"ContainerDied","Data":"9c869b32f46fa7831f92c4fbfb4dc9cb76a4016363cc10bea0645e285eae1b2e"} Jul 27 19:45:57.711152 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.711093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" event={"ID":"13ec8e0e-df8f-418e-9c91-691c7f22082e","Type":"ContainerStarted","Data":"ade7450480ddb129018832e0ca2e78a663782657d5b0f34f0bd0ad16c3849c40"} Jul 27 19:45:57.745591 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.745567 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:45:57.882587 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:57.882566 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm"] Jul 27 19:45:57.890184 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:45:57.890161 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4e3756_4ce4_4d91_b3b2_4d44008779bd.slice/crio-a860740360dcc4098963e1caf7a26f3b9601b1bcc0fd2db5c9475fb67e3363de WatchSource:0}: Error finding container a860740360dcc4098963e1caf7a26f3b9601b1bcc0fd2db5c9475fb67e3363de: Status 404 returned error can't find the container with id a860740360dcc4098963e1caf7a26f3b9601b1bcc0fd2db5c9475fb67e3363de Jul 27 19:45:58.717687 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:58.717659 2573 generic.go:356] "Generic (PLEG): container finished" podID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerID="4d1929b906f9ab3ddbae2af208c172c4cbfac9a632c13946b488b1170a11c7a0" exitCode=0 Jul 27 19:45:58.717790 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:58.717734 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" event={"ID":"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9","Type":"ContainerDied","Data":"4d1929b906f9ab3ddbae2af208c172c4cbfac9a632c13946b488b1170a11c7a0"} Jul 27 19:45:58.719174 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:58.719153 2573 generic.go:356] "Generic (PLEG): container finished" podID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerID="6b884e644a7e7ec46cacb1c90f8beda4dd8cd2b37e73ee860a82b8a8abd9022c" exitCode=0 Jul 27 19:45:58.719284 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:58.719229 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" event={"ID":"7e4e3756-4ce4-4d91-b3b2-4d44008779bd","Type":"ContainerDied","Data":"6b884e644a7e7ec46cacb1c90f8beda4dd8cd2b37e73ee860a82b8a8abd9022c"} Jul 27 19:45:58.719284 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:58.719263 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" event={"ID":"7e4e3756-4ce4-4d91-b3b2-4d44008779bd","Type":"ContainerStarted","Data":"a860740360dcc4098963e1caf7a26f3b9601b1bcc0fd2db5c9475fb67e3363de"} Jul 27 19:45:59.726679 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.726651 2573 generic.go:356] "Generic (PLEG): container finished" podID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerID="6d8d6a4c51509aa34d589ff3b28ba97f72b3544034c786f535fb47e0d81c03b3" exitCode=0 Jul 27 19:45:59.727081 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.726732 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" event={"ID":"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf","Type":"ContainerDied","Data":"6d8d6a4c51509aa34d589ff3b28ba97f72b3544034c786f535fb47e0d81c03b3"} Jul 27 19:45:59.728938 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.728908 2573 generic.go:356] "Generic (PLEG): container finished" podID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerID="ba7d5a3724d54ef52da6a3cd0b29af6cc59ae5fdb8308b5de6ebbf84d943a705" exitCode=0 Jul 27 19:45:59.729029 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.728960 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" event={"ID":"13ec8e0e-df8f-418e-9c91-691c7f22082e","Type":"ContainerDied","Data":"ba7d5a3724d54ef52da6a3cd0b29af6cc59ae5fdb8308b5de6ebbf84d943a705"} Jul 27 19:45:59.731886 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.731761 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" event={"ID":"7e4e3756-4ce4-4d91-b3b2-4d44008779bd","Type":"ContainerStarted","Data":"84b11706d249cf57d67d3ad818a8d8a09d2447404043bf8e4ac77fe1a7b8dbbb"} Jul 27 19:45:59.872085 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.872058 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:45:59.901649 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.901574 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-util\") pod \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " Jul 27 19:45:59.901857 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.901653 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-bundle\") pod \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " Jul 27 19:45:59.901857 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.901701 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl8jf\" (UniqueName: \"kubernetes.io/projected/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-kube-api-access-bl8jf\") pod \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\" (UID: \"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9\") " Jul 27 19:45:59.902432 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.902386 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-bundle" (OuterVolumeSpecName: "bundle") pod "49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" (UID: "49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:45:59.904825 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.904796 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-kube-api-access-bl8jf" (OuterVolumeSpecName: "kube-api-access-bl8jf") pod "49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" (UID: "49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9"). InnerVolumeSpecName "kube-api-access-bl8jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:45:59.907484 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:45:59.907463 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-util" (OuterVolumeSpecName: "util") pod "49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" (UID: "49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:00.002664 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.002631 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-util\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:00.002664 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.002660 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:00.002803 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.002669 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl8jf\" (UniqueName: \"kubernetes.io/projected/49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9-kube-api-access-bl8jf\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:00.738640 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.738608 2573 generic.go:356] "Generic (PLEG): container finished" podID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerID="739fe9c36fbd46de6f55ce0631b6857cb8505cca21e4c4ec1920e8eedd7c1bbc" exitCode=0 Jul 27 19:46:00.739027 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.738686 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" event={"ID":"13ec8e0e-df8f-418e-9c91-691c7f22082e","Type":"ContainerDied","Data":"739fe9c36fbd46de6f55ce0631b6857cb8505cca21e4c4ec1920e8eedd7c1bbc"} Jul 27 19:46:00.740693 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.740668 2573 generic.go:356] "Generic (PLEG): container finished" podID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerID="84b11706d249cf57d67d3ad818a8d8a09d2447404043bf8e4ac77fe1a7b8dbbb" exitCode=0 Jul 27 19:46:00.740693 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.740689 2573 generic.go:356] "Generic (PLEG): container finished" podID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerID="43ac2f59786d6eb2413e09b2c6d78029ad7e7183a1be45d8bdc56a65ef356e5f" exitCode=0 Jul 27 19:46:00.740845 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.740741 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" event={"ID":"7e4e3756-4ce4-4d91-b3b2-4d44008779bd","Type":"ContainerDied","Data":"84b11706d249cf57d67d3ad818a8d8a09d2447404043bf8e4ac77fe1a7b8dbbb"} Jul 27 19:46:00.740845 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.740781 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" event={"ID":"7e4e3756-4ce4-4d91-b3b2-4d44008779bd","Type":"ContainerDied","Data":"43ac2f59786d6eb2413e09b2c6d78029ad7e7183a1be45d8bdc56a65ef356e5f"} Jul 27 19:46:00.742471 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.742445 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" Jul 27 19:46:00.742602 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.742443 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw" event={"ID":"49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9","Type":"ContainerDied","Data":"e0672da0333959ed5a533acbb522b4c8425da2f6da8b7b36e7937f51f33ed964"} Jul 27 19:46:00.742602 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.742582 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0672da0333959ed5a533acbb522b4c8425da2f6da8b7b36e7937f51f33ed964" Jul 27 19:46:00.744622 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.744599 2573 generic.go:356] "Generic (PLEG): container finished" podID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerID="f1d205010fc4a85e42fc753025dea2ecc0fa9f02811ae2f635ee508b35288998" exitCode=0 Jul 27 19:46:00.744737 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:00.744643 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" event={"ID":"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf","Type":"ContainerDied","Data":"f1d205010fc4a85e42fc753025dea2ecc0fa9f02811ae2f635ee508b35288998"} Jul 27 19:46:01.890488 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:01.890462 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:46:01.919089 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:01.919067 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-util\") pod \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " Jul 27 19:46:01.919197 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:01.919180 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsl8l\" (UniqueName: \"kubernetes.io/projected/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-kube-api-access-nsl8l\") pod \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " Jul 27 19:46:01.919264 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:01.919251 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-bundle\") pod \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\" (UID: \"7e4e3756-4ce4-4d91-b3b2-4d44008779bd\") " Jul 27 19:46:01.919969 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:01.919876 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-bundle" (OuterVolumeSpecName: "bundle") pod "7e4e3756-4ce4-4d91-b3b2-4d44008779bd" (UID: "7e4e3756-4ce4-4d91-b3b2-4d44008779bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:01.921569 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:01.921542 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-kube-api-access-nsl8l" (OuterVolumeSpecName: "kube-api-access-nsl8l") pod "7e4e3756-4ce4-4d91-b3b2-4d44008779bd" (UID: "7e4e3756-4ce4-4d91-b3b2-4d44008779bd"). InnerVolumeSpecName "kube-api-access-nsl8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:46:01.924951 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:01.924919 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-util" (OuterVolumeSpecName: "util") pod "7e4e3756-4ce4-4d91-b3b2-4d44008779bd" (UID: "7e4e3756-4ce4-4d91-b3b2-4d44008779bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:01.948047 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:01.948028 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:46:01.950601 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:01.950586 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:46:02.020287 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.020228 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-util\") pod \"13ec8e0e-df8f-418e-9c91-691c7f22082e\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " Jul 27 19:46:02.020287 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.020259 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn57f\" (UniqueName: \"kubernetes.io/projected/13ec8e0e-df8f-418e-9c91-691c7f22082e-kube-api-access-rn57f\") pod \"13ec8e0e-df8f-418e-9c91-691c7f22082e\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " Jul 27 19:46:02.020498 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.020329 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-util\") pod \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " Jul 27 19:46:02.020498 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.020367 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-bundle\") pod \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " Jul 27 19:46:02.020498 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.020403 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7grd\" (UniqueName: \"kubernetes.io/projected/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-kube-api-access-x7grd\") pod \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\" (UID: \"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf\") " Jul 27 19:46:02.020498 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.020431 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-bundle\") pod \"13ec8e0e-df8f-418e-9c91-691c7f22082e\" (UID: \"13ec8e0e-df8f-418e-9c91-691c7f22082e\") " Jul 27 19:46:02.020744 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.020688 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsl8l\" (UniqueName: \"kubernetes.io/projected/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-kube-api-access-nsl8l\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:02.020744 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.020705 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:02.020744 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.020718 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4e3756-4ce4-4d91-b3b2-4d44008779bd-util\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:02.021061 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.021029 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-bundle" (OuterVolumeSpecName: "bundle") pod "13ec8e0e-df8f-418e-9c91-691c7f22082e" (UID: "13ec8e0e-df8f-418e-9c91-691c7f22082e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:02.021194 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.021066 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-bundle" (OuterVolumeSpecName: "bundle") pod "d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" (UID: "d341abe1-ac9b-4c10-8171-9dcdf9be1ecf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:02.023017 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.022993 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ec8e0e-df8f-418e-9c91-691c7f22082e-kube-api-access-rn57f" (OuterVolumeSpecName: "kube-api-access-rn57f") pod "13ec8e0e-df8f-418e-9c91-691c7f22082e" (UID: "13ec8e0e-df8f-418e-9c91-691c7f22082e"). InnerVolumeSpecName "kube-api-access-rn57f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:46:02.023112 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.023067 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-kube-api-access-x7grd" (OuterVolumeSpecName: "kube-api-access-x7grd") pod "d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" (UID: "d341abe1-ac9b-4c10-8171-9dcdf9be1ecf"). InnerVolumeSpecName "kube-api-access-x7grd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:46:02.026014 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.025994 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-util" (OuterVolumeSpecName: "util") pod "13ec8e0e-df8f-418e-9c91-691c7f22082e" (UID: "13ec8e0e-df8f-418e-9c91-691c7f22082e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:02.026390 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.026373 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-util" (OuterVolumeSpecName: "util") pod "d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" (UID: "d341abe1-ac9b-4c10-8171-9dcdf9be1ecf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:02.121238 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.121203 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7grd\" (UniqueName: \"kubernetes.io/projected/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-kube-api-access-x7grd\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:02.121238 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.121237 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:02.121238 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.121246 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ec8e0e-df8f-418e-9c91-691c7f22082e-util\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:02.121434 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.121255 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn57f\" (UniqueName: \"kubernetes.io/projected/13ec8e0e-df8f-418e-9c91-691c7f22082e-kube-api-access-rn57f\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:02.121434 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.121264 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-util\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:02.121434 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.121312 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d341abe1-ac9b-4c10-8171-9dcdf9be1ecf-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:02.757576 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.757543 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" Jul 27 19:46:02.757576 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.757558 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s" event={"ID":"d341abe1-ac9b-4c10-8171-9dcdf9be1ecf","Type":"ContainerDied","Data":"ae15b75be311ade4dc553e2e4f0c3bbc30b07301b125b976d8cb949bc04ed9d3"} Jul 27 19:46:02.757788 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.757598 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae15b75be311ade4dc553e2e4f0c3bbc30b07301b125b976d8cb949bc04ed9d3" Jul 27 19:46:02.759445 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.759419 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" event={"ID":"13ec8e0e-df8f-418e-9c91-691c7f22082e","Type":"ContainerDied","Data":"ade7450480ddb129018832e0ca2e78a663782657d5b0f34f0bd0ad16c3849c40"} Jul 27 19:46:02.759584 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.759461 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ade7450480ddb129018832e0ca2e78a663782657d5b0f34f0bd0ad16c3849c40" Jul 27 19:46:02.759584 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.759427 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh" Jul 27 19:46:02.761526 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.761506 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" Jul 27 19:46:02.761645 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.761536 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm" event={"ID":"7e4e3756-4ce4-4d91-b3b2-4d44008779bd","Type":"ContainerDied","Data":"a860740360dcc4098963e1caf7a26f3b9601b1bcc0fd2db5c9475fb67e3363de"} Jul 27 19:46:02.761645 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:02.761568 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a860740360dcc4098963e1caf7a26f3b9601b1bcc0fd2db5c9475fb67e3363de" Jul 27 19:46:09.085696 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:09.085645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:12.486772 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:12.486739 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:14.454820 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:14.454745 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:16.060449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:16.060415 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:16.450639 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:16.450600 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:17.901452 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:17.901407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:19.489754 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:19.489719 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:20.303290 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:20.303243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:20.406449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:20.406394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:22.291353 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:22.291322 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:23.031443 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:23.031409 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:23.446552 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:23.446522 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:24.074574 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:24.074537 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:24.134662 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:24.134629 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:31.695522 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:31.695482 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:31.811268 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:31.811231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:33.171702 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.171668 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:33.664870 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.664838 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78"] Jul 27 19:46:33.665266 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665241 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerName="pull" Jul 27 19:46:33.665266 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665261 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerName="pull" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665274 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerName="extract" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665283 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerName="extract" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665311 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerName="extract" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665319 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerName="extract" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665329 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerName="extract" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665336 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerName="extract" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665346 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerName="pull" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665353 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerName="pull" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665362 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerName="extract" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665370 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerName="extract" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665383 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerName="util" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665391 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerName="util" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665409 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerName="util" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665417 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerName="util" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665425 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerName="util" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665431 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerName="util" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665454 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerName="util" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665462 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerName="util" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665474 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerName="pull" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665482 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerName="pull" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665492 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerName="pull" Jul 27 19:46:33.665489 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665499 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerName="pull" Jul 27 19:46:33.666982 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665594 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9" containerName="extract" Jul 27 19:46:33.666982 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665608 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ec8e0e-df8f-418e-9c91-691c7f22082e" containerName="extract" Jul 27 19:46:33.666982 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665618 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d341abe1-ac9b-4c10-8171-9dcdf9be1ecf" containerName="extract" Jul 27 19:46:33.666982 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.665629 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e4e3756-4ce4-4d91-b3b2-4d44008779bd" containerName="extract" Jul 27 19:46:33.670377 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.670357 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:33.674732 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.674706 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 19:46:33.674872 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.674793 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c5b2b\"" Jul 27 19:46:33.675070 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.675055 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 19:46:33.692995 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.692969 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78"] Jul 27 19:46:33.808627 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.808593 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac2c5542-8603-409c-9d1c-a20b3f1840e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xl78\" (UID: \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:33.808810 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.808668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfsfl\" (UniqueName: \"kubernetes.io/projected/ac2c5542-8603-409c-9d1c-a20b3f1840e9-kube-api-access-sfsfl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xl78\" (UID: \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:33.909457 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.909421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac2c5542-8603-409c-9d1c-a20b3f1840e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xl78\" (UID: \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:33.909615 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.909469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfsfl\" (UniqueName: \"kubernetes.io/projected/ac2c5542-8603-409c-9d1c-a20b3f1840e9-kube-api-access-sfsfl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xl78\" (UID: \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:33.909828 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.909809 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac2c5542-8603-409c-9d1c-a20b3f1840e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xl78\" (UID: \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:33.922158 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.922103 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfsfl\" (UniqueName: \"kubernetes.io/projected/ac2c5542-8603-409c-9d1c-a20b3f1840e9-kube-api-access-sfsfl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xl78\" (UID: \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:33.981632 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:33.981610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:34.117872 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:34.117846 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78"] Jul 27 19:46:34.119571 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:46:34.119541 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2c5542_8603_409c_9d1c_a20b3f1840e9.slice/crio-3ede2f76810bc5c9dec3a44c33f0da034b5466b614cb5b0ec6db6dc66e588018 WatchSource:0}: Error finding container 3ede2f76810bc5c9dec3a44c33f0da034b5466b614cb5b0ec6db6dc66e588018: Status 404 returned error can't find the container with id 3ede2f76810bc5c9dec3a44c33f0da034b5466b614cb5b0ec6db6dc66e588018 Jul 27 19:46:34.922822 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:34.922790 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" event={"ID":"ac2c5542-8603-409c-9d1c-a20b3f1840e9","Type":"ContainerStarted","Data":"3ede2f76810bc5c9dec3a44c33f0da034b5466b614cb5b0ec6db6dc66e588018"} Jul 27 19:46:36.491454 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:36.491398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:36.599221 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:36.599171 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:38.920925 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:38.920851 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:38.951965 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:38.951935 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" event={"ID":"ac2c5542-8603-409c-9d1c-a20b3f1840e9","Type":"ContainerStarted","Data":"b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75"} Jul 27 19:46:38.952141 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:38.952113 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:38.974946 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:38.974894 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" podStartSLOduration=1.609066565 podStartE2EDuration="5.974876217s" totalImagesPullingTime="4.365809652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:38.973440077 +0000 UTC m=+403.586024663" watchObservedRunningTime="2026-07-27 19:46:38.974876217 +0000 UTC m=+403.587460800" Jul 27 19:46:39.023199 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:39.023154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:41.591433 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:41.591399 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:41.643717 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:41.643685 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:44.486716 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:44.486682 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:44.586601 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:44.586571 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:46.686093 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:46.686060 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:46.793112 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:46.793077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:48.504716 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:48.504684 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:48.604771 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:48.604741 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:49.959779 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:49.959751 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:51.376825 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:51.376790 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:51.472773 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:51.472741 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:56.170749 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:56.170708 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:56.268942 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:56.268907 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:56.956557 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:56.956520 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78"] Jul 27 19:46:56.956873 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:56.956842 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" podUID="ac2c5542-8603-409c-9d1c-a20b3f1840e9" containerName="manager" containerID="cri-o://b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75" gracePeriod=10 Jul 27 19:46:57.198228 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:57.198199 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:57.287007 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:57.286913 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfsfl\" (UniqueName: \"kubernetes.io/projected/ac2c5542-8603-409c-9d1c-a20b3f1840e9-kube-api-access-sfsfl\") pod \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\" (UID: \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\") " Jul 27 19:46:57.287007 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:57.286959 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac2c5542-8603-409c-9d1c-a20b3f1840e9-extensions-socket-volume\") pod \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\" (UID: \"ac2c5542-8603-409c-9d1c-a20b3f1840e9\") " Jul 27 19:46:57.287413 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:57.287384 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2c5542-8603-409c-9d1c-a20b3f1840e9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ac2c5542-8603-409c-9d1c-a20b3f1840e9" (UID: "ac2c5542-8603-409c-9d1c-a20b3f1840e9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:57.289208 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:57.289183 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2c5542-8603-409c-9d1c-a20b3f1840e9-kube-api-access-sfsfl" (OuterVolumeSpecName: "kube-api-access-sfsfl") pod "ac2c5542-8603-409c-9d1c-a20b3f1840e9" (UID: "ac2c5542-8603-409c-9d1c-a20b3f1840e9"). InnerVolumeSpecName "kube-api-access-sfsfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:46:57.387660 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:57.387628 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfsfl\" (UniqueName: \"kubernetes.io/projected/ac2c5542-8603-409c-9d1c-a20b3f1840e9-kube-api-access-sfsfl\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:57.387660 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:57.387653 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac2c5542-8603-409c-9d1c-a20b3f1840e9-extensions-socket-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 19:46:58.053896 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.053856 2573 generic.go:356] "Generic (PLEG): container finished" podID="ac2c5542-8603-409c-9d1c-a20b3f1840e9" containerID="b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75" exitCode=0 Jul 27 19:46:58.054066 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.053903 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" Jul 27 19:46:58.054066 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.053905 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" event={"ID":"ac2c5542-8603-409c-9d1c-a20b3f1840e9","Type":"ContainerDied","Data":"b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75"} Jul 27 19:46:58.054066 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.053964 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78" event={"ID":"ac2c5542-8603-409c-9d1c-a20b3f1840e9","Type":"ContainerDied","Data":"3ede2f76810bc5c9dec3a44c33f0da034b5466b614cb5b0ec6db6dc66e588018"} Jul 27 19:46:58.054066 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.054002 2573 scope.go:117] "RemoveContainer" containerID="b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75" Jul 27 19:46:58.062955 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.062929 2573 scope.go:117] "RemoveContainer" containerID="b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75" Jul 27 19:46:58.063189 ip-10-0-141-6 kubenswrapper[2573]: E0727 19:46:58.063171 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75\": container with ID starting with b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75 not found: ID does not exist" containerID="b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75" Jul 27 19:46:58.063256 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.063197 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75"} err="failed to get container status \"b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75\": rpc error: code = NotFound desc = could not find container \"b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75\": container with ID starting with b0df5a9a23cca0128b406eb8f7534bf57ec0f586ac82d362c01a142524fbdf75 not found: ID does not exist" Jul 27 19:46:58.078824 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.078802 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78"] Jul 27 19:46:58.089373 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.089352 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xl78"] Jul 27 19:46:58.615012 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.614976 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:58.679422 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:58.679384 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:46:59.875138 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:46:59.875099 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac2c5542-8603-409c-9d1c-a20b3f1840e9" path="/var/lib/kubelet/pods/ac2c5542-8603-409c-9d1c-a20b3f1840e9/volumes" Jul 27 19:47:04.378776 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:04.378744 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:04.450865 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:04.450830 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:11.766428 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:11.766394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:11.872238 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:11.872205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:13.283242 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:13.283208 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:13.738501 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:13.738466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:15.160030 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.159998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:15.262979 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.262951 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:15.366492 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.366462 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:15.556760 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.556732 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:47:15.557133 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.557119 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac2c5542-8603-409c-9d1c-a20b3f1840e9" containerName="manager" Jul 27 19:47:15.557196 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.557135 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac2c5542-8603-409c-9d1c-a20b3f1840e9" containerName="manager" Jul 27 19:47:15.557196 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.557195 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac2c5542-8603-409c-9d1c-a20b3f1840e9" containerName="manager" Jul 27 19:47:15.561271 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.561257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:15.563847 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.563826 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 19:47:15.563965 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.563827 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 19:47:15.564064 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.564039 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9przh\"" Jul 27 19:47:15.564194 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.564181 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 19:47:15.568485 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.568275 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:47:15.603950 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.603918 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:47:15.643500 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.643466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gpzm\" (UniqueName: \"kubernetes.io/projected/c0a79376-660e-45b2-a744-9703fc2cc8df-kube-api-access-9gpzm\") pod \"limitador-limitador-54d6d988d5-fj8gw\" (UID: \"c0a79376-660e-45b2-a744-9703fc2cc8df\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:15.643650 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.643506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0a79376-660e-45b2-a744-9703fc2cc8df-config-file\") pod \"limitador-limitador-54d6d988d5-fj8gw\" (UID: \"c0a79376-660e-45b2-a744-9703fc2cc8df\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:15.744379 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.744344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gpzm\" (UniqueName: \"kubernetes.io/projected/c0a79376-660e-45b2-a744-9703fc2cc8df-kube-api-access-9gpzm\") pod \"limitador-limitador-54d6d988d5-fj8gw\" (UID: \"c0a79376-660e-45b2-a744-9703fc2cc8df\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:15.744554 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.744386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0a79376-660e-45b2-a744-9703fc2cc8df-config-file\") pod \"limitador-limitador-54d6d988d5-fj8gw\" (UID: \"c0a79376-660e-45b2-a744-9703fc2cc8df\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:15.744975 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.744958 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0a79376-660e-45b2-a744-9703fc2cc8df-config-file\") pod \"limitador-limitador-54d6d988d5-fj8gw\" (UID: \"c0a79376-660e-45b2-a744-9703fc2cc8df\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:15.753261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.753229 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gpzm\" (UniqueName: \"kubernetes.io/projected/c0a79376-660e-45b2-a744-9703fc2cc8df-kube-api-access-9gpzm\") pod \"limitador-limitador-54d6d988d5-fj8gw\" (UID: \"c0a79376-660e-45b2-a744-9703fc2cc8df\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:15.871940 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:15.871863 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:16.038040 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:16.038014 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:47:16.040118 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:47:16.040084 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a79376_660e_45b2_a744_9703fc2cc8df.slice/crio-f17be3f3d4148baf034e270aefe9dedd32c8136564c3fc5bb741593166dbac60 WatchSource:0}: Error finding container f17be3f3d4148baf034e270aefe9dedd32c8136564c3fc5bb741593166dbac60: Status 404 returned error can't find the container with id f17be3f3d4148baf034e270aefe9dedd32c8136564c3fc5bb741593166dbac60 Jul 27 19:47:16.107137 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:16.107092 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:16.148070 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:16.147998 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" event={"ID":"c0a79376-660e-45b2-a744-9703fc2cc8df","Type":"ContainerStarted","Data":"f17be3f3d4148baf034e270aefe9dedd32c8136564c3fc5bb741593166dbac60"} Jul 27 19:47:16.162178 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:16.162154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:16.461617 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:16.461583 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:16.560383 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:16.560355 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:18.875466 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:18.875431 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:19.012785 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:19.012763 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:19.166972 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:19.166882 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" event={"ID":"c0a79376-660e-45b2-a744-9703fc2cc8df","Type":"ContainerStarted","Data":"8c95131e1e24fbf6532b8827c37737047736b0fda1a4a383d70c014930213c1d"} Jul 27 19:47:19.167118 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:19.166997 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:19.188112 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:19.188051 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" podStartSLOduration=1.2709342750000001 podStartE2EDuration="4.188035039s" totalImagesPullingTime="2.917100764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:47:19.186203574 +0000 UTC m=+443.798788176" watchObservedRunningTime="2026-07-27 19:47:19.188035039 +0000 UTC m=+443.800619622" Jul 27 19:47:30.173025 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:30.172992 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fj8gw" Jul 27 19:47:36.215589 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:36.215540 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:36.317900 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:36.317862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:36.697416 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:36.697383 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:48.502124 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:48.502088 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:48.766977 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:48.766901 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:55.123873 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:55.123841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:55.201426 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:55.201393 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:55.260270 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:55.260236 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:55.357690 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:55.357656 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:58.351971 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:58.351941 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:47:58.451250 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:47:58.451218 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:30.017789 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:30.017754 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:30.120226 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:30.120198 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:30.688013 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:30.687976 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:30.881720 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:30.881686 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:31.233876 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:31.233843 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:31.332895 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:31.332865 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:31.494095 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:31.494009 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:48:41.481052 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:41.481011 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:41.580168 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:41.580132 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:45.389261 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:45.389201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:46.854491 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:46.854407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:47.245697 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:47.245660 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:48:55.271747 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:55.271708 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:55.940907 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:55.940876 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:48:59.637206 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.637163 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm"] Jul 27 19:48:59.640745 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.640723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.643374 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.643350 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 19:48:59.643507 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.643362 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 19:48:59.644411 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.644393 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 27 19:48:59.644515 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.644457 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pvx6j\"" Jul 27 19:48:59.651756 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.651734 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm"] Jul 27 19:48:59.711732 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.711701 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbbf\" (UniqueName: \"kubernetes.io/projected/d078e159-4ecb-4a0f-b645-5ec21c4225c0-kube-api-access-tlbbf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.711910 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.711746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.711910 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.711769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.711910 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.711787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d078e159-4ecb-4a0f-b645-5ec21c4225c0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.712066 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.711917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.712066 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.711955 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.737100 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.737062 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:48:59.812839 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.812804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.812839 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.812843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.813055 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.812897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbbf\" (UniqueName: \"kubernetes.io/projected/d078e159-4ecb-4a0f-b645-5ec21c4225c0-kube-api-access-tlbbf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.813055 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.812922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.813055 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.812942 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.813055 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.812962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d078e159-4ecb-4a0f-b645-5ec21c4225c0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.813276 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.813241 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.813367 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.813288 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.813367 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.813344 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.815404 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.815368 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d078e159-4ecb-4a0f-b645-5ec21c4225c0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.815542 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.815524 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d078e159-4ecb-4a0f-b645-5ec21c4225c0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.822136 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.822116 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbbf\" (UniqueName: \"kubernetes.io/projected/d078e159-4ecb-4a0f-b645-5ec21c4225c0-kube-api-access-tlbbf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nqfm\" (UID: \"d078e159-4ecb-4a0f-b645-5ec21c4225c0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:48:59.952432 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:48:59.952399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:49:00.092581 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:00.092555 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm"] Jul 27 19:49:00.094363 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:49:00.094329 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd078e159_4ecb_4a0f_b645_5ec21c4225c0.slice/crio-ea80e6e748c2d4c64cc60a3e010065ff4da78258831b65ab016d1914e2abda4c WatchSource:0}: Error finding container ea80e6e748c2d4c64cc60a3e010065ff4da78258831b65ab016d1914e2abda4c: Status 404 returned error can't find the container with id ea80e6e748c2d4c64cc60a3e010065ff4da78258831b65ab016d1914e2abda4c Jul 27 19:49:00.681926 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:00.681889 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" event={"ID":"d078e159-4ecb-4a0f-b645-5ec21c4225c0","Type":"ContainerStarted","Data":"ea80e6e748c2d4c64cc60a3e010065ff4da78258831b65ab016d1914e2abda4c"} Jul 27 19:49:00.932449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:00.932370 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:49:01.688666 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:01.688629 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" event={"ID":"d078e159-4ecb-4a0f-b645-5ec21c4225c0","Type":"ContainerStarted","Data":"05da802aff4e973dd653ba68c0b84ac5e5d70c03dcdce2f374b059dccc56b139"} Jul 27 19:49:01.706821 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:01.706769 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" podStartSLOduration=1.688435196 podStartE2EDuration="2.706749427s" totalImagesPullingTime="1.018314231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:49:01.706042714 +0000 UTC m=+546.318627345" watchObservedRunningTime="2026-07-27 19:49:01.706749427 +0000 UTC m=+546.319334010" Jul 27 19:49:09.953541 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:09.953492 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:49:09.954054 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:09.953575 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:49:09.965824 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:09.965803 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:49:10.428941 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:10.428915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:49:10.749057 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:10.748977 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nqfm" Jul 27 19:49:18.265384 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:18.265351 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:49:19.941350 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:19.941306 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:49:34.838169 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.838129 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm"] Jul 27 19:49:34.840939 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.840920 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:34.843543 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.843523 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 27 19:49:34.851765 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.851744 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm"] Jul 27 19:49:34.917182 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.917145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:34.917182 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.917175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:34.917450 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.917218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:34.917450 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.917256 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcpb\" (UniqueName: \"kubernetes.io/projected/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-kube-api-access-2qcpb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:34.917450 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.917289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:34.917450 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.917347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:34.927428 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:34.927390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:49:35.018132 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.018103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.018339 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.018139 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.018339 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.018195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.018339 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.018219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.018339 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.018257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.018339 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.018332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcpb\" (UniqueName: \"kubernetes.io/projected/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-kube-api-access-2qcpb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.018711 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.018686 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.018711 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.018702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.018839 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.018780 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.020692 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.020670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.020935 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.020916 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.028286 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.028261 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcpb\" (UniqueName: \"kubernetes.io/projected/1b0f13f4-0fa8-4973-999b-c7e19dbbc95c-kube-api-access-2qcpb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hrqhm\" (UID: \"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.151526 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.151439 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:35.283693 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.283670 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm"] Jul 27 19:49:35.285252 ip-10-0-141-6 kubenswrapper[2573]: W0727 19:49:35.285224 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0f13f4_0fa8_4973_999b_c7e19dbbc95c.slice/crio-b721bfd8965ce4a30b7a67e12e648d3aa1853d0d5bdd8bfa4b8fa92f543530e6 WatchSource:0}: Error finding container b721bfd8965ce4a30b7a67e12e648d3aa1853d0d5bdd8bfa4b8fa92f543530e6: Status 404 returned error can't find the container with id b721bfd8965ce4a30b7a67e12e648d3aa1853d0d5bdd8bfa4b8fa92f543530e6 Jul 27 19:49:35.862096 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.862008 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" event={"ID":"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c","Type":"ContainerStarted","Data":"a8b488c89586b4aa8e944e7c07fdc9cfd7ead18156854d03a6070c2081215d77"} Jul 27 19:49:35.862096 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.862061 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" event={"ID":"1b0f13f4-0fa8-4973-999b-c7e19dbbc95c","Type":"ContainerStarted","Data":"b721bfd8965ce4a30b7a67e12e648d3aa1853d0d5bdd8bfa4b8fa92f543530e6"} Jul 27 19:49:35.881166 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:35.881111 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" podStartSLOduration=1.6992062460000001 podStartE2EDuration="1.881087858s" totalImagesPullingTime="181.881612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:49:35.879290182 +0000 UTC m=+580.491874790" watchObservedRunningTime="2026-07-27 19:49:35.881087858 +0000 UTC m=+580.493672419" Jul 27 19:49:37.936841 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:37.936809 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:49:45.152550 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:45.152514 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:45.153053 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:45.152581 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:45.164986 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:45.164965 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:45.581629 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:45.581597 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:49:45.924449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:45.924363 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hrqhm" Jul 27 19:49:49.531936 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:49:49.531897 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:50:13.014752 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:50:13.014697 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:50:14.123131 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:50:14.123094 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:50:56.980449 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:50:56.980404 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:50:57.081472 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:50:57.081421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:50:57.295348 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:50:57.295232 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:50:58.032536 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:50:58.032505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:50:58.130396 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:50:58.130366 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:51:03.682597 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:51:03.682527 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:51:03.781563 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:51:03.781515 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:51:24.598858 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:51:24.598808 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:51:24.653592 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:51:24.653560 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:51:25.039444 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:51:25.039407 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:51:35.228935 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:51:35.228901 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:51:45.836384 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:51:45.836347 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:51:51.378747 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:51:51.378699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:52:03.357620 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:52:03.357576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:52:09.021917 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:52:09.021877 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:52:09.090665 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:52:09.090632 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:02.247930 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:02.247893 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:03.531881 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:03.531845 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:30.838407 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:30.838322 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:53:38.814310 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:38.814264 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:38.913122 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:38.913083 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:39.027434 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:39.027404 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:44.832702 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:44.832643 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:44.939477 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:44.939445 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:45.031129 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:45.031068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:45.186612 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:45.186583 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:50.955790 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:50.955759 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:53:51.057210 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:53:51.057177 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:54:07.841430 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:54:07.841390 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:54:17.350568 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:54:17.350532 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:54:26.757743 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:54:26.757708 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:54:37.736008 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:54:37.735972 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:54:49.243321 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:54:49.243227 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:54:58.751087 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:54:58.751055 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:55:09.754039 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:55:09.753995 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:55:20.039030 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:55:20.038991 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:55:41.551611 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:55:41.551567 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:55:53.141905 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:55:53.141873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:56:15.847651 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:56:15.847613 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:56:27.337424 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:56:27.337385 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:56:43.432676 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:56:43.432637 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:56:53.241660 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:56:53.241618 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:56:54.508411 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:56:54.508376 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:57:02.738686 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:57:02.738646 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:57:12.339121 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:57:12.339077 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:57:22.136829 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:57:22.136797 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:59:10.837094 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:59:10.837053 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:59:19.228160 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:59:19.228083 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:59:24.684582 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:59:24.684552 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:59:32.055418 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:59:32.055381 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:59:38.555066 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:59:38.555028 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 19:59:42.852279 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:59:42.852240 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 19:59:53.551595 ip-10-0-141-6 kubenswrapper[2573]: I0727 19:59:53.551558 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:00:05.036677 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:00:05.034152 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:00:13.833108 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:00:13.833065 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:00:24.540732 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:00:24.540697 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:00:35.131568 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:00:35.131532 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:00:43.934875 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:00:43.934832 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:00:57.543059 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:00:57.542964 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:01:14.638058 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:14.638023 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:01:23.836462 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:23.836423 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:01:40.097543 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:40.097505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:01:42.558231 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:42.558191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:01:43.095056 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:43.095022 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:01:43.196457 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:43.196419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:01:43.245678 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:43.245646 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:01:43.556057 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:43.556026 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:01:49.067325 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:49.067260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:01:53.522997 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:53.522957 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76dcd887fb-h22qn"] Jul 27 20:01:53.714934 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:53.714897 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:01:53.997071 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:01:53.997030 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:02:01.028871 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:01.028815 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:02:17.892972 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:17.892876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:02:18.542667 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.542619 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76dcd887fb-h22qn" podUID="a4328e67-0b2a-4b56-b002-d2518ffffb11" containerName="console" containerID="cri-o://01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5" gracePeriod=15 Jul 27 20:02:18.794524 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.794469 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 20:02:18.804661 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.804635 2573 generic.go:356] "Generic (PLEG): container finished" podID="a4328e67-0b2a-4b56-b002-d2518ffffb11" containerID="01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5" exitCode=0 Jul 27 20:02:18.804800 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.804673 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dcd887fb-h22qn" Jul 27 20:02:18.804800 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.804705 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dcd887fb-h22qn" event={"ID":"a4328e67-0b2a-4b56-b002-d2518ffffb11","Type":"ContainerDied","Data":"01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5"} Jul 27 20:02:18.804800 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.804751 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dcd887fb-h22qn" event={"ID":"a4328e67-0b2a-4b56-b002-d2518ffffb11","Type":"ContainerDied","Data":"762832a05f16285916135b332d4cd7918691e2dea4aba04bd5d10a0643e49be1"} Jul 27 20:02:18.804800 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.804779 2573 scope.go:117] "RemoveContainer" containerID="01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5" Jul 27 20:02:18.814522 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.813589 2573 scope.go:117] "RemoveContainer" containerID="01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5" Jul 27 20:02:18.814522 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:02:18.814494 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5\": container with ID starting with 01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5 not found: ID does not exist" containerID="01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5" Jul 27 20:02:18.814646 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.814529 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5"} err="failed to get container status \"01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5\": rpc error: code = NotFound desc = could not find container \"01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5\": container with ID starting with 01f9cd8ba83a06e7a5ba8250aa3ce89075b7b431bc0995ea4eb7ff50542be4b5 not found: ID does not exist" Jul 27 20:02:18.834985 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.834954 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r99sh\" (UniqueName: \"kubernetes.io/projected/a4328e67-0b2a-4b56-b002-d2518ffffb11-kube-api-access-r99sh\") pod \"a4328e67-0b2a-4b56-b002-d2518ffffb11\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " Jul 27 20:02:18.835104 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835011 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-serving-cert\") pod \"a4328e67-0b2a-4b56-b002-d2518ffffb11\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " Jul 27 20:02:18.835104 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835044 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-config\") pod \"a4328e67-0b2a-4b56-b002-d2518ffffb11\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " Jul 27 20:02:18.835104 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835095 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-oauth-serving-cert\") pod \"a4328e67-0b2a-4b56-b002-d2518ffffb11\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " Jul 27 20:02:18.835309 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835138 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-trusted-ca-bundle\") pod \"a4328e67-0b2a-4b56-b002-d2518ffffb11\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " Jul 27 20:02:18.835309 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835167 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-oauth-config\") pod \"a4328e67-0b2a-4b56-b002-d2518ffffb11\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " Jul 27 20:02:18.835309 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835229 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-service-ca\") pod \"a4328e67-0b2a-4b56-b002-d2518ffffb11\" (UID: \"a4328e67-0b2a-4b56-b002-d2518ffffb11\") " Jul 27 20:02:18.835480 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835450 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-config" (OuterVolumeSpecName: "console-config") pod "a4328e67-0b2a-4b56-b002-d2518ffffb11" (UID: "a4328e67-0b2a-4b56-b002-d2518ffffb11"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:02:18.835561 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835542 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:02:18.835642 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835611 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a4328e67-0b2a-4b56-b002-d2518ffffb11" (UID: "a4328e67-0b2a-4b56-b002-d2518ffffb11"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:02:18.835859 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.835835 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-service-ca" (OuterVolumeSpecName: "service-ca") pod "a4328e67-0b2a-4b56-b002-d2518ffffb11" (UID: "a4328e67-0b2a-4b56-b002-d2518ffffb11"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:02:18.836046 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.836022 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a4328e67-0b2a-4b56-b002-d2518ffffb11" (UID: "a4328e67-0b2a-4b56-b002-d2518ffffb11"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:02:18.837495 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.837473 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4328e67-0b2a-4b56-b002-d2518ffffb11-kube-api-access-r99sh" (OuterVolumeSpecName: "kube-api-access-r99sh") pod "a4328e67-0b2a-4b56-b002-d2518ffffb11" (UID: "a4328e67-0b2a-4b56-b002-d2518ffffb11"). InnerVolumeSpecName "kube-api-access-r99sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:02:18.837495 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.837488 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a4328e67-0b2a-4b56-b002-d2518ffffb11" (UID: "a4328e67-0b2a-4b56-b002-d2518ffffb11"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:02:18.837676 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.837657 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a4328e67-0b2a-4b56-b002-d2518ffffb11" (UID: "a4328e67-0b2a-4b56-b002-d2518ffffb11"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:02:18.936569 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.936533 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r99sh\" (UniqueName: \"kubernetes.io/projected/a4328e67-0b2a-4b56-b002-d2518ffffb11-kube-api-access-r99sh\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:02:18.936569 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.936561 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:02:18.936569 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.936570 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-oauth-serving-cert\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:02:18.937004 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.936581 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-trusted-ca-bundle\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:02:18.937004 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.936589 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4328e67-0b2a-4b56-b002-d2518ffffb11-console-oauth-config\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:02:18.937004 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:18.936598 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4328e67-0b2a-4b56-b002-d2518ffffb11-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:02:19.128240 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:19.128202 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76dcd887fb-h22qn"] Jul 27 20:02:19.131961 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:19.131937 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76dcd887fb-h22qn"] Jul 27 20:02:19.875224 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:19.875190 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4328e67-0b2a-4b56-b002-d2518ffffb11" path="/var/lib/kubelet/pods/a4328e67-0b2a-4b56-b002-d2518ffffb11/volumes" Jul 27 20:02:20.599190 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:20.599153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:02:22.037673 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:22.037638 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:02:36.133822 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:36.133783 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:02:42.736660 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:42.736622 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:02:58.539102 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:02:58.539071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:03:10.347825 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:03:10.347791 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:03:19.536987 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:03:19.536956 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:03:27.833055 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:03:27.833021 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:03:38.734690 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:03:38.734653 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:03:47.941359 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:03:47.941261 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:04:00.190040 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:04:00.190009 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:04:00.255219 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:04:00.255179 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:04:00.934103 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:04:00.934065 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:04:10.634429 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:04:10.634393 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:04:19.946352 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:04:19.946311 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:04:29.630659 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:04:29.630622 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:04:42.934285 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:04:42.934251 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:04:51.942215 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:04:51.942179 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:05:02.686567 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:02.686533 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:05:02.918964 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:02.918920 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:05:04.038949 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:04.038918 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:05:15.565781 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:15.565742 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:05:16.634022 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:16.633983 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:05:29.735361 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:29.735322 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:05:31.994608 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:31.994568 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:05:43.240502 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:43.240469 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:05:53.039259 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:53.039220 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:05:53.337723 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:53.337608 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:05:56.350950 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:56.350915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:05:56.447896 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:56.447850 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:05:56.771830 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:05:56.771799 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:02.853515 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:02.853478 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:02.955654 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:02.955620 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:03.089939 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:03.089907 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:03.445741 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:03.445697 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:06:03.465419 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:03.465382 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:03.663289 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:03.663247 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:06.172940 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:06.172896 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:06.304176 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:06.304142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:06.399186 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:06.399150 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:06.549873 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:06.549836 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:06.876795 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:06.876703 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:06.998764 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:06.998735 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw"] Jul 27 20:06:06.999102 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:06.999081 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4328e67-0b2a-4b56-b002-d2518ffffb11" containerName="console" Jul 27 20:06:06.999102 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:06.999096 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4328e67-0b2a-4b56-b002-d2518ffffb11" containerName="console" Jul 27 20:06:06.999321 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:06.999210 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4328e67-0b2a-4b56-b002-d2518ffffb11" containerName="console" Jul 27 20:06:07.002760 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.002732 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7"] Jul 27 20:06:07.002917 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.002873 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:07.006082 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.006063 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:07.006826 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.006805 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7f6e65b7\"" Jul 27 20:06:07.016102 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.016083 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw"] Jul 27 20:06:07.023778 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.023751 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7"] Jul 27 20:06:07.102702 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.102674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6rlm\" (UniqueName: \"kubernetes.io/projected/0b745cf0-96a6-483f-beab-e538627ccf5f-kube-api-access-m6rlm\") pod \"payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7\" (UID: \"0b745cf0-96a6-483f-beab-e538627ccf5f\") " pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:07.102872 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.102712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f7786-b97c-4235-8714-ce98bbc95533-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw\" (UID: \"4e4f7786-b97c-4235-8714-ce98bbc95533\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:07.102872 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.102744 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9hb2\" (UniqueName: \"kubernetes.io/projected/4e4f7786-b97c-4235-8714-ce98bbc95533-kube-api-access-n9hb2\") pod \"payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw\" (UID: \"4e4f7786-b97c-4235-8714-ce98bbc95533\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:07.102872 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.102772 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b745cf0-96a6-483f-beab-e538627ccf5f-plugins-config-volume\") pod \"payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7\" (UID: \"0b745cf0-96a6-483f-beab-e538627ccf5f\") " pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:07.180103 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.180071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:07.204258 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.204223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6rlm\" (UniqueName: \"kubernetes.io/projected/0b745cf0-96a6-483f-beab-e538627ccf5f-kube-api-access-m6rlm\") pod \"payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7\" (UID: \"0b745cf0-96a6-483f-beab-e538627ccf5f\") " pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:07.204436 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.204276 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f7786-b97c-4235-8714-ce98bbc95533-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw\" (UID: \"4e4f7786-b97c-4235-8714-ce98bbc95533\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:07.204436 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.204352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9hb2\" (UniqueName: \"kubernetes.io/projected/4e4f7786-b97c-4235-8714-ce98bbc95533-kube-api-access-n9hb2\") pod \"payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw\" (UID: \"4e4f7786-b97c-4235-8714-ce98bbc95533\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:07.204436 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.204389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b745cf0-96a6-483f-beab-e538627ccf5f-plugins-config-volume\") pod \"payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7\" (UID: \"0b745cf0-96a6-483f-beab-e538627ccf5f\") " pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:07.205127 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.205105 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f7786-b97c-4235-8714-ce98bbc95533-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw\" (UID: \"4e4f7786-b97c-4235-8714-ce98bbc95533\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:07.205188 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.205129 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b745cf0-96a6-483f-beab-e538627ccf5f-plugins-config-volume\") pod \"payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7\" (UID: \"0b745cf0-96a6-483f-beab-e538627ccf5f\") " pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:07.214060 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.214035 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6rlm\" (UniqueName: \"kubernetes.io/projected/0b745cf0-96a6-483f-beab-e538627ccf5f-kube-api-access-m6rlm\") pod \"payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7\" (UID: \"0b745cf0-96a6-483f-beab-e538627ccf5f\") " pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:07.214162 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.214151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9hb2\" (UniqueName: \"kubernetes.io/projected/4e4f7786-b97c-4235-8714-ce98bbc95533-kube-api-access-n9hb2\") pod \"payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw\" (UID: \"4e4f7786-b97c-4235-8714-ce98bbc95533\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:07.313927 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.313885 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:07.322968 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.322946 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:07.464365 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.464331 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw"] Jul 27 20:06:07.465448 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:06:07.465420 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4f7786_b97c_4235_8714_ce98bbc95533.slice/crio-1019c4ec627b17f6c11e7814c11d5d0df267f0010cabcf60154265b402a59dd0 WatchSource:0}: Error finding container 1019c4ec627b17f6c11e7814c11d5d0df267f0010cabcf60154265b402a59dd0: Status 404 returned error can't find the container with id 1019c4ec627b17f6c11e7814c11d5d0df267f0010cabcf60154265b402a59dd0 Jul 27 20:06:07.467409 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.467387 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 20:06:07.490349 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.490327 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7"] Jul 27 20:06:07.492008 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:06:07.491986 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b745cf0_96a6_483f_beab_e538627ccf5f.slice/crio-dad00918bd5f292911a293792672442c7993af430176da43d7783eb28df6012f WatchSource:0}: Error finding container dad00918bd5f292911a293792672442c7993af430176da43d7783eb28df6012f: Status 404 returned error can't find the container with id dad00918bd5f292911a293792672442c7993af430176da43d7783eb28df6012f Jul 27 20:06:07.977655 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.977614 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" event={"ID":"0b745cf0-96a6-483f-beab-e538627ccf5f","Type":"ContainerStarted","Data":"dad00918bd5f292911a293792672442c7993af430176da43d7783eb28df6012f"} Jul 27 20:06:07.978862 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:07.978830 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" event={"ID":"4e4f7786-b97c-4235-8714-ce98bbc95533","Type":"ContainerStarted","Data":"1019c4ec627b17f6c11e7814c11d5d0df267f0010cabcf60154265b402a59dd0"} Jul 27 20:06:09.993103 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:09.993050 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" event={"ID":"4e4f7786-b97c-4235-8714-ce98bbc95533","Type":"ContainerStarted","Data":"98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312"} Jul 27 20:06:09.993632 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:09.993143 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:09.994653 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:09.994626 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" event={"ID":"0b745cf0-96a6-483f-beab-e538627ccf5f","Type":"ContainerStarted","Data":"57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1"} Jul 27 20:06:09.994784 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:09.994687 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:10.015446 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:10.015393 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" podStartSLOduration=2.374308758 podStartE2EDuration="4.015375336s" totalImagesPullingTime="1.641066578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:10.013525997 +0000 UTC m=+1574.626110605" watchObservedRunningTime="2026-07-27 20:06:10.015375336 +0000 UTC m=+1574.627959918" Jul 27 20:06:10.030906 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:10.030857 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" podStartSLOduration=2.347914094 podStartE2EDuration="4.030840974s" totalImagesPullingTime="1.68292688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:10.029998927 +0000 UTC m=+1574.642583513" watchObservedRunningTime="2026-07-27 20:06:10.030840974 +0000 UTC m=+1574.643425582" Jul 27 20:06:10.839268 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:10.839233 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:06:21.002659 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:21.002614 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" podUID="4e4f7786-b97c-4235-8714-ce98bbc95533" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.40:9004: connect: connection refused" Jul 27 20:06:21.003122 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:21.003105 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:27.315063 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:27.315022 2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" podUID="4e4f7786-b97c-4235-8714-ce98bbc95533" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.40:9004: connect: connection refused" Jul 27 20:06:27.521050 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:27.521012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:31.002739 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.002705 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:31.666884 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.666847 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm"] Jul 27 20:06:31.670354 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.670339 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" Jul 27 20:06:31.673564 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.673546 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:06:31.673650 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.673549 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:06:31.680395 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.680376 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm"] Jul 27 20:06:31.838752 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.838714 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8267f956-5b70-4b82-bc5e-499801bbede1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm\" (UID: \"8267f956-5b70-4b82-bc5e-499801bbede1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" Jul 27 20:06:31.940042 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.940011 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8267f956-5b70-4b82-bc5e-499801bbede1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm\" (UID: \"8267f956-5b70-4b82-bc5e-499801bbede1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" Jul 27 20:06:31.940634 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.940615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8267f956-5b70-4b82-bc5e-499801bbede1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm\" (UID: \"8267f956-5b70-4b82-bc5e-499801bbede1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" Jul 27 20:06:31.981939 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:31.981904 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" Jul 27 20:06:32.117024 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:32.117000 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm"] Jul 27 20:06:32.119059 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:06:32.119030 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8267f956_5b70_4b82_bc5e_499801bbede1.slice/crio-76455769d89910bc34626c66960011ac3c8d8ddf5ee4b1958a1ed5690b661ab2 WatchSource:0}: Error finding container 76455769d89910bc34626c66960011ac3c8d8ddf5ee4b1958a1ed5690b661ab2: Status 404 returned error can't find the container with id 76455769d89910bc34626c66960011ac3c8d8ddf5ee4b1958a1ed5690b661ab2 Jul 27 20:06:33.128645 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:33.128610 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" event={"ID":"8267f956-5b70-4b82-bc5e-499801bbede1","Type":"ContainerStarted","Data":"76455769d89910bc34626c66960011ac3c8d8ddf5ee4b1958a1ed5690b661ab2"} Jul 27 20:06:35.140941 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:35.140903 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" event={"ID":"8267f956-5b70-4b82-bc5e-499801bbede1","Type":"ContainerStarted","Data":"c74b66fd22da119a2f8992ead2c69a8694720a5bf4f1b8199f309b12849e8a8e"} Jul 27 20:06:35.160078 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:35.159989 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" podStartSLOduration=1.681667821 podStartE2EDuration="4.159974533s" totalImagesPullingTime="2.478306712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:35.158267924 +0000 UTC m=+1599.770852509" watchObservedRunningTime="2026-07-27 20:06:35.159974533 +0000 UTC m=+1599.772559115" Jul 27 20:06:36.147541 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:36.147502 2573 generic.go:356] "Generic (PLEG): container finished" podID="8267f956-5b70-4b82-bc5e-499801bbede1" containerID="c74b66fd22da119a2f8992ead2c69a8694720a5bf4f1b8199f309b12849e8a8e" exitCode=0 Jul 27 20:06:36.147935 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:36.147586 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" event={"ID":"8267f956-5b70-4b82-bc5e-499801bbede1","Type":"ContainerDied","Data":"c74b66fd22da119a2f8992ead2c69a8694720a5bf4f1b8199f309b12849e8a8e"} Jul 27 20:06:37.285351 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:37.285326 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" Jul 27 20:06:37.389042 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:37.389006 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8267f956-5b70-4b82-bc5e-499801bbede1-maas-api-service-ca\") pod \"8267f956-5b70-4b82-bc5e-499801bbede1\" (UID: \"8267f956-5b70-4b82-bc5e-499801bbede1\") " Jul 27 20:06:37.389424 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:37.389404 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8267f956-5b70-4b82-bc5e-499801bbede1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8267f956-5b70-4b82-bc5e-499801bbede1" (UID: "8267f956-5b70-4b82-bc5e-499801bbede1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:06:37.490359 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:37.490323 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8267f956-5b70-4b82-bc5e-499801bbede1-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:06:37.909065 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:37.909019 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:38.160276 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:38.160185 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" event={"ID":"8267f956-5b70-4b82-bc5e-499801bbede1","Type":"ContainerDied","Data":"76455769d89910bc34626c66960011ac3c8d8ddf5ee4b1958a1ed5690b661ab2"} Jul 27 20:06:38.160276 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:38.160202 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm" Jul 27 20:06:38.160276 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:38.160232 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76455769d89910bc34626c66960011ac3c8d8ddf5ee4b1958a1ed5690b661ab2" Jul 27 20:06:42.412971 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.412920 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:42.451631 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.451599 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7"] Jul 27 20:06:42.451845 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.451819 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" podUID="0b745cf0-96a6-483f-beab-e538627ccf5f" containerName="payload-processing" containerID="cri-o://57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1" gracePeriod=30 Jul 27 20:06:42.463687 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.463654 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw"] Jul 27 20:06:42.464201 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.463965 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" podUID="4e4f7786-b97c-4235-8714-ce98bbc95533" containerName="payload-pre-processing" containerID="cri-o://98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312" gracePeriod=30 Jul 27 20:06:42.509559 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.509516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:42.596110 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.596073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:42.738232 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.738209 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:42.741044 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.741022 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:42.835875 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.835829 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6rlm\" (UniqueName: \"kubernetes.io/projected/0b745cf0-96a6-483f-beab-e538627ccf5f-kube-api-access-m6rlm\") pod \"0b745cf0-96a6-483f-beab-e538627ccf5f\" (UID: \"0b745cf0-96a6-483f-beab-e538627ccf5f\") " Jul 27 20:06:42.836078 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.835922 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9hb2\" (UniqueName: \"kubernetes.io/projected/4e4f7786-b97c-4235-8714-ce98bbc95533-kube-api-access-n9hb2\") pod \"4e4f7786-b97c-4235-8714-ce98bbc95533\" (UID: \"4e4f7786-b97c-4235-8714-ce98bbc95533\") " Jul 27 20:06:42.836078 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.835941 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b745cf0-96a6-483f-beab-e538627ccf5f-plugins-config-volume\") pod \"0b745cf0-96a6-483f-beab-e538627ccf5f\" (UID: \"0b745cf0-96a6-483f-beab-e538627ccf5f\") " Jul 27 20:06:42.836078 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.835977 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f7786-b97c-4235-8714-ce98bbc95533-plugins-config-volume\") pod \"4e4f7786-b97c-4235-8714-ce98bbc95533\" (UID: \"4e4f7786-b97c-4235-8714-ce98bbc95533\") " Jul 27 20:06:42.836418 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.836391 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b745cf0-96a6-483f-beab-e538627ccf5f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b745cf0-96a6-483f-beab-e538627ccf5f" (UID: "0b745cf0-96a6-483f-beab-e538627ccf5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:06:42.836418 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.836409 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4f7786-b97c-4235-8714-ce98bbc95533-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e4f7786-b97c-4235-8714-ce98bbc95533" (UID: "4e4f7786-b97c-4235-8714-ce98bbc95533"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:06:42.838152 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.838131 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4f7786-b97c-4235-8714-ce98bbc95533-kube-api-access-n9hb2" (OuterVolumeSpecName: "kube-api-access-n9hb2") pod "4e4f7786-b97c-4235-8714-ce98bbc95533" (UID: "4e4f7786-b97c-4235-8714-ce98bbc95533"). InnerVolumeSpecName "kube-api-access-n9hb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:06:42.838152 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.838137 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b745cf0-96a6-483f-beab-e538627ccf5f-kube-api-access-m6rlm" (OuterVolumeSpecName: "kube-api-access-m6rlm") pod "0b745cf0-96a6-483f-beab-e538627ccf5f" (UID: "0b745cf0-96a6-483f-beab-e538627ccf5f"). InnerVolumeSpecName "kube-api-access-m6rlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:06:42.937227 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.937192 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6rlm\" (UniqueName: \"kubernetes.io/projected/0b745cf0-96a6-483f-beab-e538627ccf5f-kube-api-access-m6rlm\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:06:42.937227 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.937219 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9hb2\" (UniqueName: \"kubernetes.io/projected/4e4f7786-b97c-4235-8714-ce98bbc95533-kube-api-access-n9hb2\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:06:42.937227 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.937229 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b745cf0-96a6-483f-beab-e538627ccf5f-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:06:42.937512 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:42.937239 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f7786-b97c-4235-8714-ce98bbc95533-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:06:43.188755 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.188724 2573 generic.go:356] "Generic (PLEG): container finished" podID="4e4f7786-b97c-4235-8714-ce98bbc95533" containerID="98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312" exitCode=0 Jul 27 20:06:43.188941 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.188763 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" Jul 27 20:06:43.188941 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.188799 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" event={"ID":"4e4f7786-b97c-4235-8714-ce98bbc95533","Type":"ContainerDied","Data":"98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312"} Jul 27 20:06:43.188941 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.188853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw" event={"ID":"4e4f7786-b97c-4235-8714-ce98bbc95533","Type":"ContainerDied","Data":"1019c4ec627b17f6c11e7814c11d5d0df267f0010cabcf60154265b402a59dd0"} Jul 27 20:06:43.188941 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.188866 2573 scope.go:117] "RemoveContainer" containerID="98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312" Jul 27 20:06:43.190278 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.190255 2573 generic.go:356] "Generic (PLEG): container finished" podID="0b745cf0-96a6-483f-beab-e538627ccf5f" containerID="57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1" exitCode=0 Jul 27 20:06:43.190393 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.190289 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" event={"ID":"0b745cf0-96a6-483f-beab-e538627ccf5f","Type":"ContainerDied","Data":"57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1"} Jul 27 20:06:43.190393 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.190326 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" Jul 27 20:06:43.190393 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.190350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7" event={"ID":"0b745cf0-96a6-483f-beab-e538627ccf5f","Type":"ContainerDied","Data":"dad00918bd5f292911a293792672442c7993af430176da43d7783eb28df6012f"} Jul 27 20:06:43.197828 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.197769 2573 scope.go:117] "RemoveContainer" containerID="98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312" Jul 27 20:06:43.198058 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:06:43.198038 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312\": container with ID starting with 98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312 not found: ID does not exist" containerID="98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312" Jul 27 20:06:43.198129 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.198067 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312"} err="failed to get container status \"98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312\": rpc error: code = NotFound desc = could not find container \"98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312\": container with ID starting with 98253f27d70943313f94d85390bf4491702aa0efae7b50344a33e42f6d48a312 not found: ID does not exist" Jul 27 20:06:43.198129 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.198084 2573 scope.go:117] "RemoveContainer" containerID="57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1" Jul 27 20:06:43.205964 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.205948 2573 scope.go:117] "RemoveContainer" containerID="57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1" Jul 27 20:06:43.206219 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:06:43.206200 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1\": container with ID starting with 57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1 not found: ID does not exist" containerID="57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1" Jul 27 20:06:43.206275 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.206225 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1"} err="failed to get container status \"57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1\": rpc error: code = NotFound desc = could not find container \"57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1\": container with ID starting with 57a00041cff7e7de651e11360232c2235f4a222711c232fb825a1349c4b9d9a1 not found: ID does not exist" Jul 27 20:06:43.217067 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.217041 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw"] Jul 27 20:06:43.221228 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.221207 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f6e65b7-55df7bbbd4-xt5pw"] Jul 27 20:06:43.233759 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.233731 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7"] Jul 27 20:06:43.238095 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.238073 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f6e65b7-79fd8c8f48-m5pv7"] Jul 27 20:06:43.875450 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.875379 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b745cf0-96a6-483f-beab-e538627ccf5f" path="/var/lib/kubelet/pods/0b745cf0-96a6-483f-beab-e538627ccf5f/volumes" Jul 27 20:06:43.875796 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:43.875725 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e4f7786-b97c-4235-8714-ce98bbc95533" path="/var/lib/kubelet/pods/4e4f7786-b97c-4235-8714-ce98bbc95533/volumes" Jul 27 20:06:45.577260 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:45.577226 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:06:46.773586 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:46.773531 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:46.782144 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:46.782117 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm"] Jul 27 20:06:46.786380 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:46.786358 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f6e65b7-dqnbm"] Jul 27 20:06:46.965770 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:46.965740 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:47.223556 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:47.223527 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:47.473188 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:47.473146 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:47.571564 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:47.571482 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:47.669558 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:47.669523 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:47.875689 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:47.875613 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8267f956-5b70-4b82-bc5e-499801bbede1" path="/var/lib/kubelet/pods/8267f956-5b70-4b82-bc5e-499801bbede1/volumes" Jul 27 20:06:47.950757 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:47.950724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:48.046571 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:48.046535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:50.588197 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:50.588167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:50.711568 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:50.711533 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:51.039175 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.039139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:51.370584 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.370491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:51.471630 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.471599 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:06:51.523227 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523201 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4"] Jul 27 20:06:51.523626 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523609 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4f7786-b97c-4235-8714-ce98bbc95533" containerName="payload-pre-processing" Jul 27 20:06:51.523677 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523629 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4f7786-b97c-4235-8714-ce98bbc95533" containerName="payload-pre-processing" Jul 27 20:06:51.523677 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523637 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8267f956-5b70-4b82-bc5e-499801bbede1" containerName="revoke-keys" Jul 27 20:06:51.523677 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523642 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8267f956-5b70-4b82-bc5e-499801bbede1" containerName="revoke-keys" Jul 27 20:06:51.523677 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523654 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b745cf0-96a6-483f-beab-e538627ccf5f" containerName="payload-processing" Jul 27 20:06:51.523677 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523659 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b745cf0-96a6-483f-beab-e538627ccf5f" containerName="payload-processing" Jul 27 20:06:51.523830 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523720 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b745cf0-96a6-483f-beab-e538627ccf5f" containerName="payload-processing" Jul 27 20:06:51.523830 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523727 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4f7786-b97c-4235-8714-ce98bbc95533" containerName="payload-pre-processing" Jul 27 20:06:51.523830 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.523736 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8267f956-5b70-4b82-bc5e-499801bbede1" containerName="revoke-keys" Jul 27 20:06:51.529547 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.529530 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:06:51.534859 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.534833 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f658717b\"" Jul 27 20:06:51.535000 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.534833 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f658717b-dockercfg-hxbj9\"" Jul 27 20:06:51.542615 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.542592 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4"] Jul 27 20:06:51.562290 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.562249 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn"] Jul 27 20:06:51.566262 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.566239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:06:51.580179 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.580152 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn"] Jul 27 20:06:51.612648 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.612611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn\" (UID: \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:06:51.613023 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.612655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcbv\" (UniqueName: \"kubernetes.io/projected/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-kube-api-access-dgcbv\") pod \"payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4\" (UID: \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\") " pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:06:51.613023 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.612681 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6sk\" (UniqueName: \"kubernetes.io/projected/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-kube-api-access-4n6sk\") pod \"payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn\" (UID: \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:06:51.613023 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.612759 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4\" (UID: \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\") " pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:06:51.713363 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.713319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4\" (UID: \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\") " pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:06:51.713579 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.713410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn\" (UID: \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:06:51.713579 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.713444 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcbv\" (UniqueName: \"kubernetes.io/projected/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-kube-api-access-dgcbv\") pod \"payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4\" (UID: \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\") " pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:06:51.713579 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.713469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6sk\" (UniqueName: \"kubernetes.io/projected/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-kube-api-access-4n6sk\") pod \"payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn\" (UID: \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:06:51.714048 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.714026 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4\" (UID: \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\") " pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:06:51.714129 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.714111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn\" (UID: \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:06:51.724916 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.724893 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6sk\" (UniqueName: \"kubernetes.io/projected/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-kube-api-access-4n6sk\") pod \"payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn\" (UID: \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:06:51.726576 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.726557 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcbv\" (UniqueName: \"kubernetes.io/projected/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-kube-api-access-dgcbv\") pod \"payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4\" (UID: \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\") " pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:06:51.839826 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.839791 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:06:51.875806 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.875773 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:06:51.991868 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:51.991843 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4"] Jul 27 20:06:51.993583 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:06:51.993554 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f34dcb_9fb0_4014_997d_c35fa7f8c372.slice/crio-5e14067fc36ef7a57dcf52f2292109ec48e818d514e04db1c6c5b3322efaf3d1 WatchSource:0}: Error finding container 5e14067fc36ef7a57dcf52f2292109ec48e818d514e04db1c6c5b3322efaf3d1: Status 404 returned error can't find the container with id 5e14067fc36ef7a57dcf52f2292109ec48e818d514e04db1c6c5b3322efaf3d1 Jul 27 20:06:52.035532 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:06:52.035507 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2aaabd_18c9_40c9_9cbb_7c182d7f55e0.slice/crio-4a19fb35661552148cc24ae2c0ef6186cdeba9aae374e54c1d65a669d8f53a4d WatchSource:0}: Error finding container 4a19fb35661552148cc24ae2c0ef6186cdeba9aae374e54c1d65a669d8f53a4d: Status 404 returned error can't find the container with id 4a19fb35661552148cc24ae2c0ef6186cdeba9aae374e54c1d65a669d8f53a4d Jul 27 20:06:52.035934 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:52.035911 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn"] Jul 27 20:06:52.245018 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:52.244926 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" event={"ID":"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0","Type":"ContainerStarted","Data":"dcd0f5bb4189d0800f1530eef74fcfd454ca5a2654a6fba43f01a114983a8a13"} Jul 27 20:06:52.245018 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:52.244990 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:06:52.245018 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:52.245016 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" event={"ID":"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0","Type":"ContainerStarted","Data":"4a19fb35661552148cc24ae2c0ef6186cdeba9aae374e54c1d65a669d8f53a4d"} Jul 27 20:06:52.246556 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:52.246532 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" event={"ID":"b7f34dcb-9fb0-4014-997d-c35fa7f8c372","Type":"ContainerStarted","Data":"5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034"} Jul 27 20:06:52.246654 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:52.246570 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" event={"ID":"b7f34dcb-9fb0-4014-997d-c35fa7f8c372","Type":"ContainerStarted","Data":"5e14067fc36ef7a57dcf52f2292109ec48e818d514e04db1c6c5b3322efaf3d1"} Jul 27 20:06:52.246654 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:52.246589 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:06:52.263445 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:52.263396 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" podStartSLOduration=1.26338139 podStartE2EDuration="1.26338139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:52.262955145 +0000 UTC m=+1616.875539731" watchObservedRunningTime="2026-07-27 20:06:52.26338139 +0000 UTC m=+1616.875966011" Jul 27 20:06:52.283764 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:52.283718 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" podStartSLOduration=1.283705014 podStartE2EDuration="1.283705014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:52.28181779 +0000 UTC m=+1616.894402373" watchObservedRunningTime="2026-07-27 20:06:52.283705014 +0000 UTC m=+1616.896289597" Jul 27 20:06:57.057145 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:06:57.057104 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:07:03.255515 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:03.255478 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:07:03.256007 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:03.255573 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:07:15.539700 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.539669 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd"] Jul 27 20:07:15.547423 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.547392 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" Jul 27 20:07:15.550510 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.550490 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:07:15.550653 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.550569 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:07:15.555697 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.555321 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd"] Jul 27 20:07:15.635027 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.634998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df360405-1c25-470d-9f34-bbf1f4fb27b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd\" (UID: \"df360405-1c25-470d-9f34-bbf1f4fb27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" Jul 27 20:07:15.735595 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.735571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df360405-1c25-470d-9f34-bbf1f4fb27b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd\" (UID: \"df360405-1c25-470d-9f34-bbf1f4fb27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" Jul 27 20:07:15.736171 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.736153 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df360405-1c25-470d-9f34-bbf1f4fb27b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd\" (UID: \"df360405-1c25-470d-9f34-bbf1f4fb27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" Jul 27 20:07:15.859812 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.859711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" Jul 27 20:07:15.998738 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:15.998709 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd"] Jul 27 20:07:16.001284 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:07:16.001254 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf360405_1c25_470d_9f34_bbf1f4fb27b7.slice/crio-5b8dd0053b78ecba471101f067377573d25034e7d49196c91462d85ec0942956 WatchSource:0}: Error finding container 5b8dd0053b78ecba471101f067377573d25034e7d49196c91462d85ec0942956: Status 404 returned error can't find the container with id 5b8dd0053b78ecba471101f067377573d25034e7d49196c91462d85ec0942956 Jul 27 20:07:16.372612 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:16.372580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" event={"ID":"df360405-1c25-470d-9f34-bbf1f4fb27b7","Type":"ContainerStarted","Data":"e513054ad55f958e2175be75eee5d508e500cd1c5f2880896d7af5c9bbefbf93"} Jul 27 20:07:16.372787 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:16.372634 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" event={"ID":"df360405-1c25-470d-9f34-bbf1f4fb27b7","Type":"ContainerStarted","Data":"5b8dd0053b78ecba471101f067377573d25034e7d49196c91462d85ec0942956"} Jul 27 20:07:16.391821 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:16.391773 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" podStartSLOduration=1.391758707 podStartE2EDuration="1.391758707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:16.388060886 +0000 UTC m=+1641.000645469" watchObservedRunningTime="2026-07-27 20:07:16.391758707 +0000 UTC m=+1641.004343290" Jul 27 20:07:17.379201 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:17.379169 2573 generic.go:356] "Generic (PLEG): container finished" podID="df360405-1c25-470d-9f34-bbf1f4fb27b7" containerID="e513054ad55f958e2175be75eee5d508e500cd1c5f2880896d7af5c9bbefbf93" exitCode=0 Jul 27 20:07:17.379578 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:17.379255 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" event={"ID":"df360405-1c25-470d-9f34-bbf1f4fb27b7","Type":"ContainerDied","Data":"e513054ad55f958e2175be75eee5d508e500cd1c5f2880896d7af5c9bbefbf93"} Jul 27 20:07:18.512064 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:18.512038 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" Jul 27 20:07:18.664229 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:18.664143 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df360405-1c25-470d-9f34-bbf1f4fb27b7-maas-api-service-ca\") pod \"df360405-1c25-470d-9f34-bbf1f4fb27b7\" (UID: \"df360405-1c25-470d-9f34-bbf1f4fb27b7\") " Jul 27 20:07:18.664526 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:18.664503 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df360405-1c25-470d-9f34-bbf1f4fb27b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df360405-1c25-470d-9f34-bbf1f4fb27b7" (UID: "df360405-1c25-470d-9f34-bbf1f4fb27b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:07:18.765183 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:18.765146 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df360405-1c25-470d-9f34-bbf1f4fb27b7-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:07:19.391291 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:19.391244 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" event={"ID":"df360405-1c25-470d-9f34-bbf1f4fb27b7","Type":"ContainerDied","Data":"5b8dd0053b78ecba471101f067377573d25034e7d49196c91462d85ec0942956"} Jul 27 20:07:19.391520 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:19.391331 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b8dd0053b78ecba471101f067377573d25034e7d49196c91462d85ec0942956" Jul 27 20:07:19.391520 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:19.391262 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd" Jul 27 20:07:26.138338 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.138289 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:26.165270 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.165239 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4"] Jul 27 20:07:26.165555 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.165507 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" podUID="b7f34dcb-9fb0-4014-997d-c35fa7f8c372" containerName="payload-processing" containerID="cri-o://5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034" gracePeriod=30 Jul 27 20:07:26.176083 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.176057 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn"] Jul 27 20:07:26.176333 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.176290 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" podUID="0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0" containerName="payload-pre-processing" containerID="cri-o://dcd0f5bb4189d0800f1530eef74fcfd454ca5a2654a6fba43f01a114983a8a13" gracePeriod=30 Jul 27 20:07:26.419711 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.419678 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:07:26.425810 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.425783 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-plugins-config-volume\") pod \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\" (UID: \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\") " Jul 27 20:07:26.425968 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.425878 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgcbv\" (UniqueName: \"kubernetes.io/projected/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-kube-api-access-dgcbv\") pod \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\" (UID: \"b7f34dcb-9fb0-4014-997d-c35fa7f8c372\") " Jul 27 20:07:26.426166 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.426140 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7f34dcb-9fb0-4014-997d-c35fa7f8c372" (UID: "b7f34dcb-9fb0-4014-997d-c35fa7f8c372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:07:26.428134 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.428110 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-kube-api-access-dgcbv" (OuterVolumeSpecName: "kube-api-access-dgcbv") pod "b7f34dcb-9fb0-4014-997d-c35fa7f8c372" (UID: "b7f34dcb-9fb0-4014-997d-c35fa7f8c372"). InnerVolumeSpecName "kube-api-access-dgcbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:07:26.429728 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.429701 2573 generic.go:356] "Generic (PLEG): container finished" podID="b7f34dcb-9fb0-4014-997d-c35fa7f8c372" containerID="5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034" exitCode=0 Jul 27 20:07:26.429826 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.429746 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" Jul 27 20:07:26.429826 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.429777 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" event={"ID":"b7f34dcb-9fb0-4014-997d-c35fa7f8c372","Type":"ContainerDied","Data":"5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034"} Jul 27 20:07:26.429944 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.429827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4" event={"ID":"b7f34dcb-9fb0-4014-997d-c35fa7f8c372","Type":"ContainerDied","Data":"5e14067fc36ef7a57dcf52f2292109ec48e818d514e04db1c6c5b3322efaf3d1"} Jul 27 20:07:26.429944 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.429851 2573 scope.go:117] "RemoveContainer" containerID="5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034" Jul 27 20:07:26.431766 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.431733 2573 generic.go:356] "Generic (PLEG): container finished" podID="0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0" containerID="dcd0f5bb4189d0800f1530eef74fcfd454ca5a2654a6fba43f01a114983a8a13" exitCode=0 Jul 27 20:07:26.431860 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.431801 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" event={"ID":"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0","Type":"ContainerDied","Data":"dcd0f5bb4189d0800f1530eef74fcfd454ca5a2654a6fba43f01a114983a8a13"} Jul 27 20:07:26.446148 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.446132 2573 scope.go:117] "RemoveContainer" containerID="5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034" Jul 27 20:07:26.446425 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:07:26.446404 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034\": container with ID starting with 5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034 not found: ID does not exist" containerID="5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034" Jul 27 20:07:26.446490 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.446438 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034"} err="failed to get container status \"5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034\": rpc error: code = NotFound desc = could not find container \"5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034\": container with ID starting with 5c45e92e2f24214f7063cffbaf365e72f84788dae8e818afc1a1011962aab034 not found: ID does not exist" Jul 27 20:07:26.450538 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.450521 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:07:26.462506 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.462364 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4"] Jul 27 20:07:26.466154 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.466133 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f658717b-6cfbc6bbd9-qgbd4"] Jul 27 20:07:26.526897 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.526868 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n6sk\" (UniqueName: \"kubernetes.io/projected/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-kube-api-access-4n6sk\") pod \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\" (UID: \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\") " Jul 27 20:07:26.527049 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.526985 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-plugins-config-volume\") pod \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\" (UID: \"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0\") " Jul 27 20:07:26.527247 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.527231 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgcbv\" (UniqueName: \"kubernetes.io/projected/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-kube-api-access-dgcbv\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:07:26.527287 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.527255 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f34dcb-9fb0-4014-997d-c35fa7f8c372-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:07:26.527344 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.527327 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0" (UID: "0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:07:26.529067 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.529049 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-kube-api-access-4n6sk" (OuterVolumeSpecName: "kube-api-access-4n6sk") pod "0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0" (UID: "0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0"). InnerVolumeSpecName "kube-api-access-4n6sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:07:26.628673 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.628645 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:07:26.628673 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:26.628670 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n6sk\" (UniqueName: \"kubernetes.io/projected/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0-kube-api-access-4n6sk\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:07:27.438559 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:27.438481 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" Jul 27 20:07:27.438559 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:27.438529 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn" event={"ID":"0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0","Type":"ContainerDied","Data":"4a19fb35661552148cc24ae2c0ef6186cdeba9aae374e54c1d65a669d8f53a4d"} Jul 27 20:07:27.439035 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:27.438590 2573 scope.go:117] "RemoveContainer" containerID="dcd0f5bb4189d0800f1530eef74fcfd454ca5a2654a6fba43f01a114983a8a13" Jul 27 20:07:27.463173 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:27.463152 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn"] Jul 27 20:07:27.465248 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:27.465226 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f658717b-74c4bbd95b-2vfmn"] Jul 27 20:07:27.875623 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:27.875542 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0" path="/var/lib/kubelet/pods/0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0/volumes" Jul 27 20:07:27.875909 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:27.875896 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f34dcb-9fb0-4014-997d-c35fa7f8c372" path="/var/lib/kubelet/pods/b7f34dcb-9fb0-4014-997d-c35fa7f8c372/volumes" Jul 27 20:07:27.896580 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:27.896554 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:29.140487 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:29.140449 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:07:30.614456 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:30.614415 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:30.615774 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:30.615750 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd"] Jul 27 20:07:30.617796 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:30.617769 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f658717b-vh4vd"] Jul 27 20:07:30.767720 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:30.767686 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:31.003889 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:31.003855 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:31.225881 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:31.225849 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:31.545016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:31.544984 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:31.782700 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:31.782665 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:31.875566 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:31.875469 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df360405-1c25-470d-9f34-bbf1f4fb27b7" path="/var/lib/kubelet/pods/df360405-1c25-470d-9f34-bbf1f4fb27b7/volumes" Jul 27 20:07:33.742979 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:33.742941 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:33.838000 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:33.837958 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:33.949594 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:33.949556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:34.182951 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.182918 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:34.552489 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.552395 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:34.652123 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.652090 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:07:34.656430 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656408 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd"] Jul 27 20:07:34.656751 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656739 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df360405-1c25-470d-9f34-bbf1f4fb27b7" containerName="revoke-keys" Jul 27 20:07:34.656794 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656753 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df360405-1c25-470d-9f34-bbf1f4fb27b7" containerName="revoke-keys" Jul 27 20:07:34.656794 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656776 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f34dcb-9fb0-4014-997d-c35fa7f8c372" containerName="payload-processing" Jul 27 20:07:34.656794 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656782 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f34dcb-9fb0-4014-997d-c35fa7f8c372" containerName="payload-processing" Jul 27 20:07:34.656794 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656789 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0" containerName="payload-pre-processing" Jul 27 20:07:34.656794 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656794 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0" containerName="payload-pre-processing" Jul 27 20:07:34.657003 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656846 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d2aaabd-18c9-40c9-9cbb-7c182d7f55e0" containerName="payload-pre-processing" Jul 27 20:07:34.657003 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656853 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f34dcb-9fb0-4014-997d-c35fa7f8c372" containerName="payload-processing" Jul 27 20:07:34.657003 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.656861 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="df360405-1c25-470d-9f34-bbf1f4fb27b7" containerName="revoke-keys" Jul 27 20:07:34.661196 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.661178 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:34.663941 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.663924 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e3c0fa8a\"" Jul 27 20:07:34.664061 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.663929 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e3c0fa8a-dockercfg-flgnj\"" Jul 27 20:07:34.672857 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.672834 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd"] Jul 27 20:07:34.689264 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.689237 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64"] Jul 27 20:07:34.692746 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.692728 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:34.704393 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.704371 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64"] Jul 27 20:07:34.706643 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.706617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlq6t\" (UniqueName: \"kubernetes.io/projected/56279bdd-fb52-48dd-a2e4-e62b5fa65426-kube-api-access-xlq6t\") pod \"payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd\" (UID: \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:34.706760 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.706663 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56279bdd-fb52-48dd-a2e4-e62b5fa65426-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd\" (UID: \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:34.807236 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.807166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56279bdd-fb52-48dd-a2e4-e62b5fa65426-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd\" (UID: \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:34.807236 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.807213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-plugins-config-volume\") pod \"payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64\" (UID: \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\") " pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:34.807661 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.807316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7knnj\" (UniqueName: \"kubernetes.io/projected/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-kube-api-access-7knnj\") pod \"payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64\" (UID: \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\") " pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:34.807661 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.807382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlq6t\" (UniqueName: \"kubernetes.io/projected/56279bdd-fb52-48dd-a2e4-e62b5fa65426-kube-api-access-xlq6t\") pod \"payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd\" (UID: \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:34.807973 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.807955 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56279bdd-fb52-48dd-a2e4-e62b5fa65426-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd\" (UID: \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:34.816535 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.816514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlq6t\" (UniqueName: \"kubernetes.io/projected/56279bdd-fb52-48dd-a2e4-e62b5fa65426-kube-api-access-xlq6t\") pod \"payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd\" (UID: \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:34.908847 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.908811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-plugins-config-volume\") pod \"payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64\" (UID: \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\") " pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:34.909044 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.908909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7knnj\" (UniqueName: \"kubernetes.io/projected/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-kube-api-access-7knnj\") pod \"payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64\" (UID: \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\") " pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:34.909649 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.909627 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-plugins-config-volume\") pod \"payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64\" (UID: \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\") " pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:34.917624 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.917597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7knnj\" (UniqueName: \"kubernetes.io/projected/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-kube-api-access-7knnj\") pod \"payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64\" (UID: \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\") " pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:34.971282 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:34.971250 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:35.002901 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.002871 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:35.128085 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:07:35.128057 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56279bdd_fb52_48dd_a2e4_e62b5fa65426.slice/crio-53b71f267dfa177fb45aaa0d88edf18329148f71ece4837ec79f8a577a7036d3 WatchSource:0}: Error finding container 53b71f267dfa177fb45aaa0d88edf18329148f71ece4837ec79f8a577a7036d3: Status 404 returned error can't find the container with id 53b71f267dfa177fb45aaa0d88edf18329148f71ece4837ec79f8a577a7036d3 Jul 27 20:07:35.128244 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.128181 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd"] Jul 27 20:07:35.143854 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.143831 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64"] Jul 27 20:07:35.145248 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:07:35.145173 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d04c9e8_e8a7_4b45_a0e1_15cf66585389.slice/crio-c17d188bfa97ade7a768778955549547f6c55a83278a355695c2c33679e79f15 WatchSource:0}: Error finding container c17d188bfa97ade7a768778955549547f6c55a83278a355695c2c33679e79f15: Status 404 returned error can't find the container with id c17d188bfa97ade7a768778955549547f6c55a83278a355695c2c33679e79f15 Jul 27 20:07:35.483171 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.483141 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" event={"ID":"8d04c9e8-e8a7-4b45-a0e1-15cf66585389","Type":"ContainerStarted","Data":"d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842"} Jul 27 20:07:35.483384 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.483245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" event={"ID":"8d04c9e8-e8a7-4b45-a0e1-15cf66585389","Type":"ContainerStarted","Data":"c17d188bfa97ade7a768778955549547f6c55a83278a355695c2c33679e79f15"} Jul 27 20:07:35.483384 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.483322 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:35.484667 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.484642 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" event={"ID":"56279bdd-fb52-48dd-a2e4-e62b5fa65426","Type":"ContainerStarted","Data":"b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68"} Jul 27 20:07:35.484757 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.484691 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" event={"ID":"56279bdd-fb52-48dd-a2e4-e62b5fa65426","Type":"ContainerStarted","Data":"53b71f267dfa177fb45aaa0d88edf18329148f71ece4837ec79f8a577a7036d3"} Jul 27 20:07:35.484757 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.484716 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:35.508590 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.508535 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" podStartSLOduration=1.508513496 podStartE2EDuration="1.508513496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:35.504875389 +0000 UTC m=+1660.117460001" watchObservedRunningTime="2026-07-27 20:07:35.508513496 +0000 UTC m=+1660.121098081" Jul 27 20:07:35.523774 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:35.523727 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" podStartSLOduration=1.523711786 podStartE2EDuration="1.523711786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:35.521726844 +0000 UTC m=+1660.134311429" watchObservedRunningTime="2026-07-27 20:07:35.523711786 +0000 UTC m=+1660.136296369" Jul 27 20:07:40.639428 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:40.639393 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:07:46.492391 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:46.492356 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:07:46.492794 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:46.492423 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:07:59.509371 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.509336 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l"] Jul 27 20:07:59.513800 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.513783 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" Jul 27 20:07:59.516689 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.516671 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:07:59.516804 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.516673 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:07:59.523202 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.523179 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l"] Jul 27 20:07:59.630263 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.630230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9785e37d-4aee-4671-93f7-41aa48efb199-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l\" (UID: \"9785e37d-4aee-4671-93f7-41aa48efb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" Jul 27 20:07:59.730895 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.730860 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9785e37d-4aee-4671-93f7-41aa48efb199-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l\" (UID: \"9785e37d-4aee-4671-93f7-41aa48efb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" Jul 27 20:07:59.731528 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.731508 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9785e37d-4aee-4671-93f7-41aa48efb199-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l\" (UID: \"9785e37d-4aee-4671-93f7-41aa48efb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" Jul 27 20:07:59.825318 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.825212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" Jul 27 20:07:59.969506 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:07:59.969484 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l"] Jul 27 20:07:59.971533 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:07:59.971506 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9785e37d_4aee_4671_93f7_41aa48efb199.slice/crio-abf5e6509ddc40c6d5aaed0627f2f9897a0541fea10cf161af5e1d0fddc36e2c WatchSource:0}: Error finding container abf5e6509ddc40c6d5aaed0627f2f9897a0541fea10cf161af5e1d0fddc36e2c: Status 404 returned error can't find the container with id abf5e6509ddc40c6d5aaed0627f2f9897a0541fea10cf161af5e1d0fddc36e2c Jul 27 20:08:00.618433 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:00.618337 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" event={"ID":"9785e37d-4aee-4671-93f7-41aa48efb199","Type":"ContainerStarted","Data":"2da3e1a0ab03bb963e8dd1c89d6c8caf30b8323c3e0f294b676f923a445c4638"} Jul 27 20:08:00.618433 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:00.618406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" event={"ID":"9785e37d-4aee-4671-93f7-41aa48efb199","Type":"ContainerStarted","Data":"abf5e6509ddc40c6d5aaed0627f2f9897a0541fea10cf161af5e1d0fddc36e2c"} Jul 27 20:08:00.635679 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:00.635636 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" podStartSLOduration=1.635621402 podStartE2EDuration="1.635621402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:00.633791472 +0000 UTC m=+1685.246376076" watchObservedRunningTime="2026-07-27 20:08:00.635621402 +0000 UTC m=+1685.248205984" Jul 27 20:08:01.625568 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:01.625534 2573 generic.go:356] "Generic (PLEG): container finished" podID="9785e37d-4aee-4671-93f7-41aa48efb199" containerID="2da3e1a0ab03bb963e8dd1c89d6c8caf30b8323c3e0f294b676f923a445c4638" exitCode=0 Jul 27 20:08:01.625969 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:01.625625 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" event={"ID":"9785e37d-4aee-4671-93f7-41aa48efb199","Type":"ContainerDied","Data":"2da3e1a0ab03bb963e8dd1c89d6c8caf30b8323c3e0f294b676f923a445c4638"} Jul 27 20:08:02.762368 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:02.762345 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" Jul 27 20:08:02.858637 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:02.858602 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9785e37d-4aee-4671-93f7-41aa48efb199-maas-api-service-ca\") pod \"9785e37d-4aee-4671-93f7-41aa48efb199\" (UID: \"9785e37d-4aee-4671-93f7-41aa48efb199\") " Jul 27 20:08:02.858970 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:02.858946 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9785e37d-4aee-4671-93f7-41aa48efb199-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9785e37d-4aee-4671-93f7-41aa48efb199" (UID: "9785e37d-4aee-4671-93f7-41aa48efb199"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:08:02.959721 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:02.959689 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9785e37d-4aee-4671-93f7-41aa48efb199-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:03.637633 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:03.637605 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" Jul 27 20:08:03.637782 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:03.637604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l" event={"ID":"9785e37d-4aee-4671-93f7-41aa48efb199","Type":"ContainerDied","Data":"abf5e6509ddc40c6d5aaed0627f2f9897a0541fea10cf161af5e1d0fddc36e2c"} Jul 27 20:08:03.637782 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:03.637703 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf5e6509ddc40c6d5aaed0627f2f9897a0541fea10cf161af5e1d0fddc36e2c" Jul 27 20:08:10.021822 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.021784 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:10.049151 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.049118 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64"] Jul 27 20:08:10.049391 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.049364 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" podUID="8d04c9e8-e8a7-4b45-a0e1-15cf66585389" containerName="payload-processing" containerID="cri-o://d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842" gracePeriod=30 Jul 27 20:08:10.057907 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.057881 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd"] Jul 27 20:08:10.058141 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.058097 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" podUID="56279bdd-fb52-48dd-a2e4-e62b5fa65426" containerName="payload-pre-processing" containerID="cri-o://b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68" gracePeriod=30 Jul 27 20:08:10.097105 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.097063 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:10.196144 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.196089 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:10.327141 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.327116 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:08:10.329966 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.329946 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:08:10.427076 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.427046 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56279bdd-fb52-48dd-a2e4-e62b5fa65426-plugins-config-volume\") pod \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\" (UID: \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\") " Jul 27 20:08:10.427253 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.427084 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-plugins-config-volume\") pod \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\" (UID: \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\") " Jul 27 20:08:10.427253 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.427113 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlq6t\" (UniqueName: \"kubernetes.io/projected/56279bdd-fb52-48dd-a2e4-e62b5fa65426-kube-api-access-xlq6t\") pod \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\" (UID: \"56279bdd-fb52-48dd-a2e4-e62b5fa65426\") " Jul 27 20:08:10.427253 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.427235 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7knnj\" (UniqueName: \"kubernetes.io/projected/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-kube-api-access-7knnj\") pod \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\" (UID: \"8d04c9e8-e8a7-4b45-a0e1-15cf66585389\") " Jul 27 20:08:10.427495 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.427470 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56279bdd-fb52-48dd-a2e4-e62b5fa65426-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56279bdd-fb52-48dd-a2e4-e62b5fa65426" (UID: "56279bdd-fb52-48dd-a2e4-e62b5fa65426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:08:10.427565 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.427506 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d04c9e8-e8a7-4b45-a0e1-15cf66585389" (UID: "8d04c9e8-e8a7-4b45-a0e1-15cf66585389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:08:10.429339 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.429317 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56279bdd-fb52-48dd-a2e4-e62b5fa65426-kube-api-access-xlq6t" (OuterVolumeSpecName: "kube-api-access-xlq6t") pod "56279bdd-fb52-48dd-a2e4-e62b5fa65426" (UID: "56279bdd-fb52-48dd-a2e4-e62b5fa65426"). InnerVolumeSpecName "kube-api-access-xlq6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:08:10.429458 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.429440 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-kube-api-access-7knnj" (OuterVolumeSpecName: "kube-api-access-7knnj") pod "8d04c9e8-e8a7-4b45-a0e1-15cf66585389" (UID: "8d04c9e8-e8a7-4b45-a0e1-15cf66585389"). InnerVolumeSpecName "kube-api-access-7knnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:08:10.528740 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.528710 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:10.528740 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.528735 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlq6t\" (UniqueName: \"kubernetes.io/projected/56279bdd-fb52-48dd-a2e4-e62b5fa65426-kube-api-access-xlq6t\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:10.528740 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.528744 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7knnj\" (UniqueName: \"kubernetes.io/projected/8d04c9e8-e8a7-4b45-a0e1-15cf66585389-kube-api-access-7knnj\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:10.528952 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.528752 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56279bdd-fb52-48dd-a2e4-e62b5fa65426-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:10.679349 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.679318 2573 generic.go:356] "Generic (PLEG): container finished" podID="8d04c9e8-e8a7-4b45-a0e1-15cf66585389" containerID="d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842" exitCode=0 Jul 27 20:08:10.679518 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.679359 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" Jul 27 20:08:10.679518 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.679390 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" event={"ID":"8d04c9e8-e8a7-4b45-a0e1-15cf66585389","Type":"ContainerDied","Data":"d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842"} Jul 27 20:08:10.679518 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.679438 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64" event={"ID":"8d04c9e8-e8a7-4b45-a0e1-15cf66585389","Type":"ContainerDied","Data":"c17d188bfa97ade7a768778955549547f6c55a83278a355695c2c33679e79f15"} Jul 27 20:08:10.679518 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.679465 2573 scope.go:117] "RemoveContainer" containerID="d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842" Jul 27 20:08:10.680758 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.680735 2573 generic.go:356] "Generic (PLEG): container finished" podID="56279bdd-fb52-48dd-a2e4-e62b5fa65426" containerID="b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68" exitCode=0 Jul 27 20:08:10.680839 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.680783 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" Jul 27 20:08:10.680878 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.680779 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" event={"ID":"56279bdd-fb52-48dd-a2e4-e62b5fa65426","Type":"ContainerDied","Data":"b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68"} Jul 27 20:08:10.680921 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.680885 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd" event={"ID":"56279bdd-fb52-48dd-a2e4-e62b5fa65426","Type":"ContainerDied","Data":"53b71f267dfa177fb45aaa0d88edf18329148f71ece4837ec79f8a577a7036d3"} Jul 27 20:08:10.688609 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.688581 2573 scope.go:117] "RemoveContainer" containerID="d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842" Jul 27 20:08:10.688861 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:08:10.688838 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842\": container with ID starting with d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842 not found: ID does not exist" containerID="d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842" Jul 27 20:08:10.688929 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.688871 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842"} err="failed to get container status \"d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842\": rpc error: code = NotFound desc = could not find container \"d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842\": container with ID starting with d4b70bd3d5b3ccebd058732d68d54a73174bb7345818a27e9c1870f2666f6842 not found: ID does not exist" Jul 27 20:08:10.688929 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.688899 2573 scope.go:117] "RemoveContainer" containerID="b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68" Jul 27 20:08:10.696617 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.696602 2573 scope.go:117] "RemoveContainer" containerID="b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68" Jul 27 20:08:10.696831 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:08:10.696815 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68\": container with ID starting with b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68 not found: ID does not exist" containerID="b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68" Jul 27 20:08:10.696879 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.696837 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68"} err="failed to get container status \"b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68\": rpc error: code = NotFound desc = could not find container \"b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68\": container with ID starting with b088504cc3d733a4d45ae669b90ceca72dd0bb9263e0d9bffa6c7cb54234ff68 not found: ID does not exist" Jul 27 20:08:10.704155 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.704136 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd"] Jul 27 20:08:10.713337 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.708989 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3c0fa8a-788c9dcf68-99kdd"] Jul 27 20:08:10.721034 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.721011 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64"] Jul 27 20:08:10.724321 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:10.724282 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3c0fa8a-77dcfd8bfd-xrs64"] Jul 27 20:08:11.875966 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:11.875932 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56279bdd-fb52-48dd-a2e4-e62b5fa65426" path="/var/lib/kubelet/pods/56279bdd-fb52-48dd-a2e4-e62b5fa65426/volumes" Jul 27 20:08:11.876361 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:11.876346 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d04c9e8-e8a7-4b45-a0e1-15cf66585389" path="/var/lib/kubelet/pods/8d04c9e8-e8a7-4b45-a0e1-15cf66585389/volumes" Jul 27 20:08:13.341828 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:13.341790 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:08:14.585775 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:14.585740 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l"] Jul 27 20:08:14.589579 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:14.589538 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:14.589918 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:14.589900 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3c0fa8a-4592l"] Jul 27 20:08:15.815153 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:15.815116 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:15.875576 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:15.875544 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9785e37d-4aee-4671-93f7-41aa48efb199" path="/var/lib/kubelet/pods/9785e37d-4aee-4671-93f7-41aa48efb199/volumes" Jul 27 20:08:16.034880 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:16.034790 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:16.271228 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:16.271192 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:16.601338 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:16.601233 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:16.981822 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:16.981788 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:19.006645 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:19.006602 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:19.112702 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:19.112669 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:19.241457 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:19.241430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:19.411057 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:19.410948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:19.934269 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:19.934234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:20.036597 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.036561 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:20.049560 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049536 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2"] Jul 27 20:08:20.049885 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049874 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56279bdd-fb52-48dd-a2e4-e62b5fa65426" containerName="payload-pre-processing" Jul 27 20:08:20.049937 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049887 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="56279bdd-fb52-48dd-a2e4-e62b5fa65426" containerName="payload-pre-processing" Jul 27 20:08:20.049937 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049899 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9785e37d-4aee-4671-93f7-41aa48efb199" containerName="revoke-keys" Jul 27 20:08:20.049937 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049904 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9785e37d-4aee-4671-93f7-41aa48efb199" containerName="revoke-keys" Jul 27 20:08:20.049937 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049914 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d04c9e8-e8a7-4b45-a0e1-15cf66585389" containerName="payload-processing" Jul 27 20:08:20.049937 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049920 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d04c9e8-e8a7-4b45-a0e1-15cf66585389" containerName="payload-processing" Jul 27 20:08:20.050094 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049972 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="56279bdd-fb52-48dd-a2e4-e62b5fa65426" containerName="payload-pre-processing" Jul 27 20:08:20.050094 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049980 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9785e37d-4aee-4671-93f7-41aa48efb199" containerName="revoke-keys" Jul 27 20:08:20.050094 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.049989 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d04c9e8-e8a7-4b45-a0e1-15cf66585389" containerName="payload-processing" Jul 27 20:08:20.054608 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.054590 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:20.057664 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.057645 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-269943c6-dockercfg-5js2s\"" Jul 27 20:08:20.057802 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.057645 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-269943c6\"" Jul 27 20:08:20.071960 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.071931 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n"] Jul 27 20:08:20.075936 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.075919 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2"] Jul 27 20:08:20.076016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.076004 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:20.088939 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.088906 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n"] Jul 27 20:08:20.112130 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.112099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ddde6-897b-43c3-a55c-e7f0fed7985a-plugins-config-volume\") pod \"payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n\" (UID: \"618ddde6-897b-43c3-a55c-e7f0fed7985a\") " pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:20.112285 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.112146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jws8k\" (UniqueName: \"kubernetes.io/projected/65449dc4-f2f5-47ea-860f-37654ba6c88e-kube-api-access-jws8k\") pod \"payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2\" (UID: \"65449dc4-f2f5-47ea-860f-37654ba6c88e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:20.112285 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.112172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ht2\" (UniqueName: \"kubernetes.io/projected/618ddde6-897b-43c3-a55c-e7f0fed7985a-kube-api-access-56ht2\") pod \"payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n\" (UID: \"618ddde6-897b-43c3-a55c-e7f0fed7985a\") " pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:20.112285 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.112240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65449dc4-f2f5-47ea-860f-37654ba6c88e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2\" (UID: \"65449dc4-f2f5-47ea-860f-37654ba6c88e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:20.213698 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.213608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ddde6-897b-43c3-a55c-e7f0fed7985a-plugins-config-volume\") pod \"payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n\" (UID: \"618ddde6-897b-43c3-a55c-e7f0fed7985a\") " pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:20.213698 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.213666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jws8k\" (UniqueName: \"kubernetes.io/projected/65449dc4-f2f5-47ea-860f-37654ba6c88e-kube-api-access-jws8k\") pod \"payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2\" (UID: \"65449dc4-f2f5-47ea-860f-37654ba6c88e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:20.213934 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.213706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ht2\" (UniqueName: \"kubernetes.io/projected/618ddde6-897b-43c3-a55c-e7f0fed7985a-kube-api-access-56ht2\") pod \"payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n\" (UID: \"618ddde6-897b-43c3-a55c-e7f0fed7985a\") " pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:20.213934 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.213746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65449dc4-f2f5-47ea-860f-37654ba6c88e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2\" (UID: \"65449dc4-f2f5-47ea-860f-37654ba6c88e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:20.214319 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.214279 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ddde6-897b-43c3-a55c-e7f0fed7985a-plugins-config-volume\") pod \"payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n\" (UID: \"618ddde6-897b-43c3-a55c-e7f0fed7985a\") " pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:20.214512 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.214493 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65449dc4-f2f5-47ea-860f-37654ba6c88e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2\" (UID: \"65449dc4-f2f5-47ea-860f-37654ba6c88e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:20.222760 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.222735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ht2\" (UniqueName: \"kubernetes.io/projected/618ddde6-897b-43c3-a55c-e7f0fed7985a-kube-api-access-56ht2\") pod \"payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n\" (UID: \"618ddde6-897b-43c3-a55c-e7f0fed7985a\") " pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:20.222760 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.222747 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jws8k\" (UniqueName: \"kubernetes.io/projected/65449dc4-f2f5-47ea-860f-37654ba6c88e-kube-api-access-jws8k\") pod \"payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2\" (UID: \"65449dc4-f2f5-47ea-860f-37654ba6c88e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:20.364992 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.364962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:20.386148 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.386122 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:20.547595 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.547571 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2"] Jul 27 20:08:20.549352 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:08:20.549327 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65449dc4_f2f5_47ea_860f_37654ba6c88e.slice/crio-4868477fe61dee57af7b774e2df4d13968054f2edda6ae4a12bc33b2d7a31676 WatchSource:0}: Error finding container 4868477fe61dee57af7b774e2df4d13968054f2edda6ae4a12bc33b2d7a31676: Status 404 returned error can't find the container with id 4868477fe61dee57af7b774e2df4d13968054f2edda6ae4a12bc33b2d7a31676 Jul 27 20:08:20.562187 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.562166 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n"] Jul 27 20:08:20.566310 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:08:20.566273 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618ddde6_897b_43c3_a55c_e7f0fed7985a.slice/crio-50336a112a069392f78631affac776d7541f344fae7ec736fa232137bb45f966 WatchSource:0}: Error finding container 50336a112a069392f78631affac776d7541f344fae7ec736fa232137bb45f966: Status 404 returned error can't find the container with id 50336a112a069392f78631affac776d7541f344fae7ec736fa232137bb45f966 Jul 27 20:08:20.740048 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.739949 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" event={"ID":"618ddde6-897b-43c3-a55c-e7f0fed7985a","Type":"ContainerStarted","Data":"e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda"} Jul 27 20:08:20.740217 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.740116 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:20.740217 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.740184 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" event={"ID":"618ddde6-897b-43c3-a55c-e7f0fed7985a","Type":"ContainerStarted","Data":"50336a112a069392f78631affac776d7541f344fae7ec736fa232137bb45f966"} Jul 27 20:08:20.741722 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.741696 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" event={"ID":"65449dc4-f2f5-47ea-860f-37654ba6c88e","Type":"ContainerStarted","Data":"e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe"} Jul 27 20:08:20.741833 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.741743 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" event={"ID":"65449dc4-f2f5-47ea-860f-37654ba6c88e","Type":"ContainerStarted","Data":"4868477fe61dee57af7b774e2df4d13968054f2edda6ae4a12bc33b2d7a31676"} Jul 27 20:08:20.741833 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.741769 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:20.758616 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.758573 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" podStartSLOduration=0.758558806 podStartE2EDuration="758.558806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:20.757195567 +0000 UTC m=+1705.369780173" watchObservedRunningTime="2026-07-27 20:08:20.758558806 +0000 UTC m=+1705.371143368" Jul 27 20:08:20.775849 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:20.775805 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" podStartSLOduration=0.775792247 podStartE2EDuration="775.792247ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:20.773643719 +0000 UTC m=+1705.386228302" watchObservedRunningTime="2026-07-27 20:08:20.775792247 +0000 UTC m=+1705.388376829" Jul 27 20:08:24.833825 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:24.833791 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:08:31.748466 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:31.748429 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:31.748973 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:31.748596 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:43.844396 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:43.844362 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q"] Jul 27 20:08:43.847929 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:43.847914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" Jul 27 20:08:43.850651 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:43.850625 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:08:43.850791 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:43.850706 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:08:43.857010 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:43.856988 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q"] Jul 27 20:08:44.023708 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:44.023680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777dd8c-3622-43b5-8963-7274ea8c70c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-269943c6-xsn7q\" (UID: \"a777dd8c-3622-43b5-8963-7274ea8c70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" Jul 27 20:08:44.125119 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:44.125034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777dd8c-3622-43b5-8963-7274ea8c70c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-269943c6-xsn7q\" (UID: \"a777dd8c-3622-43b5-8963-7274ea8c70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" Jul 27 20:08:44.125690 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:44.125673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777dd8c-3622-43b5-8963-7274ea8c70c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-269943c6-xsn7q\" (UID: \"a777dd8c-3622-43b5-8963-7274ea8c70c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" Jul 27 20:08:44.160028 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:44.160007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" Jul 27 20:08:44.294503 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:44.294477 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q"] Jul 27 20:08:44.296052 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:08:44.296027 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda777dd8c_3622_43b5_8963_7274ea8c70c0.slice/crio-53551a09f2db65e69f52970ccd8ba2b0bc78834cc78bb5045aec2cc124be78e5 WatchSource:0}: Error finding container 53551a09f2db65e69f52970ccd8ba2b0bc78834cc78bb5045aec2cc124be78e5: Status 404 returned error can't find the container with id 53551a09f2db65e69f52970ccd8ba2b0bc78834cc78bb5045aec2cc124be78e5 Jul 27 20:08:44.871848 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:44.871765 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" event={"ID":"a777dd8c-3622-43b5-8963-7274ea8c70c0","Type":"ContainerStarted","Data":"1e4b55bf18d963775e10a3ed5311b2de60176dc040acd80be62fca98fdf46b0f"} Jul 27 20:08:44.871848 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:44.871814 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" event={"ID":"a777dd8c-3622-43b5-8963-7274ea8c70c0","Type":"ContainerStarted","Data":"53551a09f2db65e69f52970ccd8ba2b0bc78834cc78bb5045aec2cc124be78e5"} Jul 27 20:08:44.889672 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:44.889622 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" podStartSLOduration=1.889610637 podStartE2EDuration="1.889610637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:44.887953777 +0000 UTC m=+1729.500538378" watchObservedRunningTime="2026-07-27 20:08:44.889610637 +0000 UTC m=+1729.502195266" Jul 27 20:08:45.880451 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:45.880405 2573 generic.go:356] "Generic (PLEG): container finished" podID="a777dd8c-3622-43b5-8963-7274ea8c70c0" containerID="1e4b55bf18d963775e10a3ed5311b2de60176dc040acd80be62fca98fdf46b0f" exitCode=0 Jul 27 20:08:45.881110 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:45.880744 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" event={"ID":"a777dd8c-3622-43b5-8963-7274ea8c70c0","Type":"ContainerDied","Data":"1e4b55bf18d963775e10a3ed5311b2de60176dc040acd80be62fca98fdf46b0f"} Jul 27 20:08:47.014526 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:47.014502 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" Jul 27 20:08:47.152961 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:47.152865 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777dd8c-3622-43b5-8963-7274ea8c70c0-maas-api-service-ca\") pod \"a777dd8c-3622-43b5-8963-7274ea8c70c0\" (UID: \"a777dd8c-3622-43b5-8963-7274ea8c70c0\") " Jul 27 20:08:47.153239 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:47.153214 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a777dd8c-3622-43b5-8963-7274ea8c70c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a777dd8c-3622-43b5-8963-7274ea8c70c0" (UID: "a777dd8c-3622-43b5-8963-7274ea8c70c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:08:47.254145 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:47.254115 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777dd8c-3622-43b5-8963-7274ea8c70c0-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:47.892932 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:47.892904 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" event={"ID":"a777dd8c-3622-43b5-8963-7274ea8c70c0","Type":"ContainerDied","Data":"53551a09f2db65e69f52970ccd8ba2b0bc78834cc78bb5045aec2cc124be78e5"} Jul 27 20:08:47.893076 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:47.892946 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53551a09f2db65e69f52970ccd8ba2b0bc78834cc78bb5045aec2cc124be78e5" Jul 27 20:08:47.893076 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:47.892914 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q" Jul 27 20:08:54.462069 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.462030 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:54.500169 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.500136 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n"] Jul 27 20:08:54.500577 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.500396 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" podUID="618ddde6-897b-43c3-a55c-e7f0fed7985a" containerName="payload-processing" containerID="cri-o://e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda" gracePeriod=30 Jul 27 20:08:54.508346 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.508282 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2"] Jul 27 20:08:54.508742 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.508571 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" podUID="65449dc4-f2f5-47ea-860f-37654ba6c88e" containerName="payload-pre-processing" containerID="cri-o://e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe" gracePeriod=30 Jul 27 20:08:54.545153 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.545127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:54.614648 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.614599 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:54.749010 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.748984 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:54.780879 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.780855 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:54.815780 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.815756 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65449dc4-f2f5-47ea-860f-37654ba6c88e-plugins-config-volume\") pod \"65449dc4-f2f5-47ea-860f-37654ba6c88e\" (UID: \"65449dc4-f2f5-47ea-860f-37654ba6c88e\") " Jul 27 20:08:54.815915 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.815847 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jws8k\" (UniqueName: \"kubernetes.io/projected/65449dc4-f2f5-47ea-860f-37654ba6c88e-kube-api-access-jws8k\") pod \"65449dc4-f2f5-47ea-860f-37654ba6c88e\" (UID: \"65449dc4-f2f5-47ea-860f-37654ba6c88e\") " Jul 27 20:08:54.815915 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.815882 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56ht2\" (UniqueName: \"kubernetes.io/projected/618ddde6-897b-43c3-a55c-e7f0fed7985a-kube-api-access-56ht2\") pod \"618ddde6-897b-43c3-a55c-e7f0fed7985a\" (UID: \"618ddde6-897b-43c3-a55c-e7f0fed7985a\") " Jul 27 20:08:54.815915 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.815909 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ddde6-897b-43c3-a55c-e7f0fed7985a-plugins-config-volume\") pod \"618ddde6-897b-43c3-a55c-e7f0fed7985a\" (UID: \"618ddde6-897b-43c3-a55c-e7f0fed7985a\") " Jul 27 20:08:54.816262 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.816125 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65449dc4-f2f5-47ea-860f-37654ba6c88e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65449dc4-f2f5-47ea-860f-37654ba6c88e" (UID: "65449dc4-f2f5-47ea-860f-37654ba6c88e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:08:54.816410 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.816283 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618ddde6-897b-43c3-a55c-e7f0fed7985a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "618ddde6-897b-43c3-a55c-e7f0fed7985a" (UID: "618ddde6-897b-43c3-a55c-e7f0fed7985a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:08:54.818598 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.818576 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618ddde6-897b-43c3-a55c-e7f0fed7985a-kube-api-access-56ht2" (OuterVolumeSpecName: "kube-api-access-56ht2") pod "618ddde6-897b-43c3-a55c-e7f0fed7985a" (UID: "618ddde6-897b-43c3-a55c-e7f0fed7985a"). InnerVolumeSpecName "kube-api-access-56ht2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:08:54.818719 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.818702 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65449dc4-f2f5-47ea-860f-37654ba6c88e-kube-api-access-jws8k" (OuterVolumeSpecName: "kube-api-access-jws8k") pod "65449dc4-f2f5-47ea-860f-37654ba6c88e" (UID: "65449dc4-f2f5-47ea-860f-37654ba6c88e"). InnerVolumeSpecName "kube-api-access-jws8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:08:54.917347 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.917155 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65449dc4-f2f5-47ea-860f-37654ba6c88e-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:54.917347 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.917185 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jws8k\" (UniqueName: \"kubernetes.io/projected/65449dc4-f2f5-47ea-860f-37654ba6c88e-kube-api-access-jws8k\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:54.917347 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.917201 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56ht2\" (UniqueName: \"kubernetes.io/projected/618ddde6-897b-43c3-a55c-e7f0fed7985a-kube-api-access-56ht2\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:54.917347 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.917216 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ddde6-897b-43c3-a55c-e7f0fed7985a-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:08:54.931865 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.931836 2573 generic.go:356] "Generic (PLEG): container finished" podID="65449dc4-f2f5-47ea-860f-37654ba6c88e" containerID="e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe" exitCode=0 Jul 27 20:08:54.931973 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.931877 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" Jul 27 20:08:54.931973 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.931920 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" event={"ID":"65449dc4-f2f5-47ea-860f-37654ba6c88e","Type":"ContainerDied","Data":"e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe"} Jul 27 20:08:54.931973 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.931965 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2" event={"ID":"65449dc4-f2f5-47ea-860f-37654ba6c88e","Type":"ContainerDied","Data":"4868477fe61dee57af7b774e2df4d13968054f2edda6ae4a12bc33b2d7a31676"} Jul 27 20:08:54.932128 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.931988 2573 scope.go:117] "RemoveContainer" containerID="e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe" Jul 27 20:08:54.933374 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.933353 2573 generic.go:356] "Generic (PLEG): container finished" podID="618ddde6-897b-43c3-a55c-e7f0fed7985a" containerID="e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda" exitCode=0 Jul 27 20:08:54.933481 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.933392 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" Jul 27 20:08:54.933481 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.933422 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" event={"ID":"618ddde6-897b-43c3-a55c-e7f0fed7985a","Type":"ContainerDied","Data":"e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda"} Jul 27 20:08:54.933481 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.933462 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n" event={"ID":"618ddde6-897b-43c3-a55c-e7f0fed7985a","Type":"ContainerDied","Data":"50336a112a069392f78631affac776d7541f344fae7ec736fa232137bb45f966"} Jul 27 20:08:54.943729 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.943706 2573 scope.go:117] "RemoveContainer" containerID="e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe" Jul 27 20:08:54.943995 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:08:54.943977 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe\": container with ID starting with e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe not found: ID does not exist" containerID="e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe" Jul 27 20:08:54.944074 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.944004 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe"} err="failed to get container status \"e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe\": rpc error: code = NotFound desc = could not find container \"e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe\": container with ID starting with e10653bd2903b814dd65a94202198dcc2b7f9518bb07080defa0d9c0a9df4ebe not found: ID does not exist" Jul 27 20:08:54.944074 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.944023 2573 scope.go:117] "RemoveContainer" containerID="e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda" Jul 27 20:08:54.952260 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.952246 2573 scope.go:117] "RemoveContainer" containerID="e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda" Jul 27 20:08:54.952507 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:08:54.952489 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda\": container with ID starting with e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda not found: ID does not exist" containerID="e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda" Jul 27 20:08:54.952564 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.952513 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda"} err="failed to get container status \"e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda\": rpc error: code = NotFound desc = could not find container \"e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda\": container with ID starting with e7ebd9c4a0c502f4f47f12ce3169d37d1fb8d752d5fb28b5bb1e16e8a6015fda not found: ID does not exist" Jul 27 20:08:54.963333 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.963277 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2"] Jul 27 20:08:54.965382 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.965364 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-269943c6-686b7b8d88-ptgf2"] Jul 27 20:08:54.977306 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.977276 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n"] Jul 27 20:08:54.982055 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:54.982005 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-269943c6-5d4746d47d-bbr4n"] Jul 27 20:08:55.878251 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:55.878089 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="618ddde6-897b-43c3-a55c-e7f0fed7985a" path="/var/lib/kubelet/pods/618ddde6-897b-43c3-a55c-e7f0fed7985a/volumes" Jul 27 20:08:55.878726 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:55.878666 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65449dc4-f2f5-47ea-860f-37654ba6c88e" path="/var/lib/kubelet/pods/65449dc4-f2f5-47ea-860f-37654ba6c88e/volumes" Jul 27 20:08:57.738761 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:57.738725 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:08:58.932350 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:58.932285 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q"] Jul 27 20:08:58.934527 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:58.934504 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-269943c6-xsn7q"] Jul 27 20:08:58.936827 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:58.936799 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:59.084424 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:59.084394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:59.369681 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:59.369596 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:59.853257 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:59.853222 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:08:59.875721 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:59.875692 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a777dd8c-3622-43b5-8963-7274ea8c70c0" path="/var/lib/kubelet/pods/a777dd8c-3622-43b5-8963-7274ea8c70c0/volumes" Jul 27 20:08:59.941917 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:08:59.941879 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:00.040378 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:00.040347 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:01.456166 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:01.456131 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:08.327717 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:08.327676 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:08.455192 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:08.455160 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:08.575128 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:08.575101 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:08.674651 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:08.674563 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:09.059189 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:09.059157 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:16.676148 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:16.676102 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:35.010640 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:35.010604 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:39.360807 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:39.360771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:41.496150 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:41.496110 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:41.597552 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:41.597515 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:41.701205 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:41.701179 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:09:57.578530 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:09:57.578495 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:03.544521 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:03.544483 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:03.621278 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:03.621249 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:03.725326 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:03.725269 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:03.824234 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:03.824191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:14.272392 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:14.272354 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:15.070536 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:15.070494 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:22.491909 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:22.491872 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:22.590988 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:22.590950 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:22.688451 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:22.688418 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:22.788127 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:22.788032 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:23.597931 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:23.597885 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:23.694053 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:23.694017 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:23.794845 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:23.794811 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:25.343715 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:25.343683 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:26.537794 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:26.537762 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:10:31.607063 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:31.607028 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:31.709840 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:31.709806 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:32.300186 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:32.300143 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:38.143896 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:38.143861 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:10:38.987242 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:38.987209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:39.089000 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:39.088962 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:39.210026 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:39.209994 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:39.558041 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:39.558005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:39.645995 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:39.645965 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:42.373913 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:42.373875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:42.469665 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:42.469632 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:42.559590 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:42.559555 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:42.937989 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:42.937948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:43.116941 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.116870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:43.217691 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.217605 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:43.319233 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319201 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd"] Jul 27 20:10:43.319598 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319585 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65449dc4-f2f5-47ea-860f-37654ba6c88e" containerName="payload-pre-processing" Jul 27 20:10:43.319644 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319599 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="65449dc4-f2f5-47ea-860f-37654ba6c88e" containerName="payload-pre-processing" Jul 27 20:10:43.319644 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319607 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a777dd8c-3622-43b5-8963-7274ea8c70c0" containerName="revoke-keys" Jul 27 20:10:43.319644 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319612 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a777dd8c-3622-43b5-8963-7274ea8c70c0" containerName="revoke-keys" Jul 27 20:10:43.319644 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319620 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618ddde6-897b-43c3-a55c-e7f0fed7985a" containerName="payload-processing" Jul 27 20:10:43.319644 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319626 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="618ddde6-897b-43c3-a55c-e7f0fed7985a" containerName="payload-processing" Jul 27 20:10:43.319799 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319676 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="618ddde6-897b-43c3-a55c-e7f0fed7985a" containerName="payload-processing" Jul 27 20:10:43.319799 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319683 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a777dd8c-3622-43b5-8963-7274ea8c70c0" containerName="revoke-keys" Jul 27 20:10:43.319799 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.319690 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="65449dc4-f2f5-47ea-860f-37654ba6c88e" containerName="payload-pre-processing" Jul 27 20:10:43.322766 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.322750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:43.325664 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.325637 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-452dddb4-dockercfg-vd8tm\"" Jul 27 20:10:43.325664 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.325638 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-452dddb4\"" Jul 27 20:10:43.332977 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.332951 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd"] Jul 27 20:10:43.375446 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.375414 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b"] Jul 27 20:10:43.379487 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.379465 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:43.380715 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.380694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hrm\" (UniqueName: \"kubernetes.io/projected/0632ec6a-1247-47cd-b436-8c3281b21eda-kube-api-access-z6hrm\") pod \"payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd\" (UID: \"0632ec6a-1247-47cd-b436-8c3281b21eda\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:43.380853 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.380757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632ec6a-1247-47cd-b436-8c3281b21eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd\" (UID: \"0632ec6a-1247-47cd-b436-8c3281b21eda\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:43.390239 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.390213 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b"] Jul 27 20:10:43.482172 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.482081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hrm\" (UniqueName: \"kubernetes.io/projected/0632ec6a-1247-47cd-b436-8c3281b21eda-kube-api-access-z6hrm\") pod \"payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd\" (UID: \"0632ec6a-1247-47cd-b436-8c3281b21eda\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:43.482172 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.482159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632ec6a-1247-47cd-b436-8c3281b21eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd\" (UID: \"0632ec6a-1247-47cd-b436-8c3281b21eda\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:43.482447 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.482194 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c291c5-dacc-406a-b637-7c86940848f1-plugins-config-volume\") pod \"payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b\" (UID: \"34c291c5-dacc-406a-b637-7c86940848f1\") " pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:43.482447 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.482214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8s9v\" (UniqueName: \"kubernetes.io/projected/34c291c5-dacc-406a-b637-7c86940848f1-kube-api-access-m8s9v\") pod \"payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b\" (UID: \"34c291c5-dacc-406a-b637-7c86940848f1\") " pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:43.482853 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.482834 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632ec6a-1247-47cd-b436-8c3281b21eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd\" (UID: \"0632ec6a-1247-47cd-b436-8c3281b21eda\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:43.491346 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.491325 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hrm\" (UniqueName: \"kubernetes.io/projected/0632ec6a-1247-47cd-b436-8c3281b21eda-kube-api-access-z6hrm\") pod \"payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd\" (UID: \"0632ec6a-1247-47cd-b436-8c3281b21eda\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:43.583240 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.583212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c291c5-dacc-406a-b637-7c86940848f1-plugins-config-volume\") pod \"payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b\" (UID: \"34c291c5-dacc-406a-b637-7c86940848f1\") " pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:43.583240 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.583244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8s9v\" (UniqueName: \"kubernetes.io/projected/34c291c5-dacc-406a-b637-7c86940848f1-kube-api-access-m8s9v\") pod \"payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b\" (UID: \"34c291c5-dacc-406a-b637-7c86940848f1\") " pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:43.584143 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.584120 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c291c5-dacc-406a-b637-7c86940848f1-plugins-config-volume\") pod \"payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b\" (UID: \"34c291c5-dacc-406a-b637-7c86940848f1\") " pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:43.593702 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.593674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8s9v\" (UniqueName: \"kubernetes.io/projected/34c291c5-dacc-406a-b637-7c86940848f1-kube-api-access-m8s9v\") pod \"payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b\" (UID: \"34c291c5-dacc-406a-b637-7c86940848f1\") " pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:43.633879 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.633848 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:43.690633 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.690603 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:43.780250 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.780100 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd"] Jul 27 20:10:43.782678 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:10:43.782624 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0632ec6a_1247_47cd_b436_8c3281b21eda.slice/crio-eeb5639891fdd4ce8001179f6f1abda23fb62e5aea815be381f920b2b1ecf44d WatchSource:0}: Error finding container eeb5639891fdd4ce8001179f6f1abda23fb62e5aea815be381f920b2b1ecf44d: Status 404 returned error can't find the container with id eeb5639891fdd4ce8001179f6f1abda23fb62e5aea815be381f920b2b1ecf44d Jul 27 20:10:43.833195 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:43.833173 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b"] Jul 27 20:10:43.836414 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:10:43.836389 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c291c5_dacc_406a_b637_7c86940848f1.slice/crio-55bc330609b240aa7b8ea223d9569550ba7e3c68e228fa154468a75d600be4b9 WatchSource:0}: Error finding container 55bc330609b240aa7b8ea223d9569550ba7e3c68e228fa154468a75d600be4b9: Status 404 returned error can't find the container with id 55bc330609b240aa7b8ea223d9569550ba7e3c68e228fa154468a75d600be4b9 Jul 27 20:10:44.508656 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:44.508623 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" event={"ID":"34c291c5-dacc-406a-b637-7c86940848f1","Type":"ContainerStarted","Data":"06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149"} Jul 27 20:10:44.509117 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:44.508682 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" event={"ID":"34c291c5-dacc-406a-b637-7c86940848f1","Type":"ContainerStarted","Data":"55bc330609b240aa7b8ea223d9569550ba7e3c68e228fa154468a75d600be4b9"} Jul 27 20:10:44.509117 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:44.508812 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:44.510273 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:44.510251 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" event={"ID":"0632ec6a-1247-47cd-b436-8c3281b21eda","Type":"ContainerStarted","Data":"4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e"} Jul 27 20:10:44.510442 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:44.510287 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" event={"ID":"0632ec6a-1247-47cd-b436-8c3281b21eda","Type":"ContainerStarted","Data":"eeb5639891fdd4ce8001179f6f1abda23fb62e5aea815be381f920b2b1ecf44d"} Jul 27 20:10:44.510442 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:44.510340 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:44.529618 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:44.529571 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" podStartSLOduration=1.529553424 podStartE2EDuration="1.529553424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:10:44.528243061 +0000 UTC m=+1849.140827847" watchObservedRunningTime="2026-07-27 20:10:44.529553424 +0000 UTC m=+1849.142138006" Jul 27 20:10:44.545968 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:44.545918 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" podStartSLOduration=1.545903256 podStartE2EDuration="1.545903256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:10:44.545522702 +0000 UTC m=+1849.158107287" watchObservedRunningTime="2026-07-27 20:10:44.545903256 +0000 UTC m=+1849.158487839" Jul 27 20:10:46.873194 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:46.873161 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:47.563198 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:47.563152 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:47.667090 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:47.667051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:47.961895 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:47.961850 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:51.160011 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:51.159970 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:10:54.340064 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.340024 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:54.458956 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.458918 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:10:54.654689 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.654610 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf"] Jul 27 20:10:54.658347 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.658330 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" Jul 27 20:10:54.661263 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.661241 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:10:54.661650 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.661629 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:10:54.669902 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.669875 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf"] Jul 27 20:10:54.786254 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.786210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b813fb9-5b95-45c7-906f-0e74a8b82ac4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf\" (UID: \"7b813fb9-5b95-45c7-906f-0e74a8b82ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" Jul 27 20:10:54.887136 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.887093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b813fb9-5b95-45c7-906f-0e74a8b82ac4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf\" (UID: \"7b813fb9-5b95-45c7-906f-0e74a8b82ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" Jul 27 20:10:54.887797 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.887774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b813fb9-5b95-45c7-906f-0e74a8b82ac4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf\" (UID: \"7b813fb9-5b95-45c7-906f-0e74a8b82ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" Jul 27 20:10:54.969572 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:54.969527 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" Jul 27 20:10:55.116640 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:55.116608 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf"] Jul 27 20:10:55.119444 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:10:55.119417 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b813fb9_5b95_45c7_906f_0e74a8b82ac4.slice/crio-51f9e86fbf3730fa80a4b99de20a1448a011749525ee7a23261fb2b41c28c1b5 WatchSource:0}: Error finding container 51f9e86fbf3730fa80a4b99de20a1448a011749525ee7a23261fb2b41c28c1b5: Status 404 returned error can't find the container with id 51f9e86fbf3730fa80a4b99de20a1448a011749525ee7a23261fb2b41c28c1b5 Jul 27 20:10:55.517660 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:55.517625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:10:55.518026 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:55.517803 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:10:55.574976 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:55.574880 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" event={"ID":"7b813fb9-5b95-45c7-906f-0e74a8b82ac4","Type":"ContainerStarted","Data":"55d4ccfd8b364f72af6f23a6e28fe82c3b9488fa264099e7989d9f057e4e544a"} Jul 27 20:10:55.574976 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:55.574939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" event={"ID":"7b813fb9-5b95-45c7-906f-0e74a8b82ac4","Type":"ContainerStarted","Data":"51f9e86fbf3730fa80a4b99de20a1448a011749525ee7a23261fb2b41c28c1b5"} Jul 27 20:10:56.581806 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:56.581775 2573 generic.go:356] "Generic (PLEG): container finished" podID="7b813fb9-5b95-45c7-906f-0e74a8b82ac4" containerID="55d4ccfd8b364f72af6f23a6e28fe82c3b9488fa264099e7989d9f057e4e544a" exitCode=0 Jul 27 20:10:56.582181 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:56.581867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" event={"ID":"7b813fb9-5b95-45c7-906f-0e74a8b82ac4","Type":"ContainerDied","Data":"55d4ccfd8b364f72af6f23a6e28fe82c3b9488fa264099e7989d9f057e4e544a"} Jul 27 20:10:57.717944 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:57.717919 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" Jul 27 20:10:57.814847 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:57.814815 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b813fb9-5b95-45c7-906f-0e74a8b82ac4-maas-api-service-ca\") pod \"7b813fb9-5b95-45c7-906f-0e74a8b82ac4\" (UID: \"7b813fb9-5b95-45c7-906f-0e74a8b82ac4\") " Jul 27 20:10:57.815231 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:57.815209 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b813fb9-5b95-45c7-906f-0e74a8b82ac4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b813fb9-5b95-45c7-906f-0e74a8b82ac4" (UID: "7b813fb9-5b95-45c7-906f-0e74a8b82ac4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:10:57.915773 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:57.915745 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b813fb9-5b95-45c7-906f-0e74a8b82ac4-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:10:58.593535 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:58.593504 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" Jul 27 20:10:58.593713 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:58.593503 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf" event={"ID":"7b813fb9-5b95-45c7-906f-0e74a8b82ac4","Type":"ContainerDied","Data":"51f9e86fbf3730fa80a4b99de20a1448a011749525ee7a23261fb2b41c28c1b5"} Jul 27 20:10:58.593713 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:10:58.593634 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51f9e86fbf3730fa80a4b99de20a1448a011749525ee7a23261fb2b41c28c1b5" Jul 27 20:11:04.938093 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:04.938051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:04.959355 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:04.959324 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b"] Jul 27 20:11:04.959599 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:04.959574 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" podUID="34c291c5-dacc-406a-b637-7c86940848f1" containerName="payload-processing" containerID="cri-o://06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149" gracePeriod=30 Jul 27 20:11:04.966768 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:04.966740 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd"] Jul 27 20:11:04.967004 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:04.966979 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" podUID="0632ec6a-1247-47cd-b436-8c3281b21eda" containerName="payload-pre-processing" containerID="cri-o://4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e" gracePeriod=30 Jul 27 20:11:04.998803 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:04.998769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:05.057039 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.056990 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:05.215984 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.215959 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:11:05.246635 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.246611 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:11:05.386616 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.386573 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8s9v\" (UniqueName: \"kubernetes.io/projected/34c291c5-dacc-406a-b637-7c86940848f1-kube-api-access-m8s9v\") pod \"34c291c5-dacc-406a-b637-7c86940848f1\" (UID: \"34c291c5-dacc-406a-b637-7c86940848f1\") " Jul 27 20:11:05.386616 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.386625 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632ec6a-1247-47cd-b436-8c3281b21eda-plugins-config-volume\") pod \"0632ec6a-1247-47cd-b436-8c3281b21eda\" (UID: \"0632ec6a-1247-47cd-b436-8c3281b21eda\") " Jul 27 20:11:05.386890 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.386648 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6hrm\" (UniqueName: \"kubernetes.io/projected/0632ec6a-1247-47cd-b436-8c3281b21eda-kube-api-access-z6hrm\") pod \"0632ec6a-1247-47cd-b436-8c3281b21eda\" (UID: \"0632ec6a-1247-47cd-b436-8c3281b21eda\") " Jul 27 20:11:05.386890 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.386691 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c291c5-dacc-406a-b637-7c86940848f1-plugins-config-volume\") pod \"34c291c5-dacc-406a-b637-7c86940848f1\" (UID: \"34c291c5-dacc-406a-b637-7c86940848f1\") " Jul 27 20:11:05.387080 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.387055 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0632ec6a-1247-47cd-b436-8c3281b21eda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0632ec6a-1247-47cd-b436-8c3281b21eda" (UID: "0632ec6a-1247-47cd-b436-8c3281b21eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:11:05.387162 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.387109 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c291c5-dacc-406a-b637-7c86940848f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34c291c5-dacc-406a-b637-7c86940848f1" (UID: "34c291c5-dacc-406a-b637-7c86940848f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:11:05.388948 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.388927 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0632ec6a-1247-47cd-b436-8c3281b21eda-kube-api-access-z6hrm" (OuterVolumeSpecName: "kube-api-access-z6hrm") pod "0632ec6a-1247-47cd-b436-8c3281b21eda" (UID: "0632ec6a-1247-47cd-b436-8c3281b21eda"). InnerVolumeSpecName "kube-api-access-z6hrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:11:05.389005 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.388932 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c291c5-dacc-406a-b637-7c86940848f1-kube-api-access-m8s9v" (OuterVolumeSpecName: "kube-api-access-m8s9v") pod "34c291c5-dacc-406a-b637-7c86940848f1" (UID: "34c291c5-dacc-406a-b637-7c86940848f1"). InnerVolumeSpecName "kube-api-access-m8s9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:11:05.487779 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.487691 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c291c5-dacc-406a-b637-7c86940848f1-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:11:05.487779 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.487721 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8s9v\" (UniqueName: \"kubernetes.io/projected/34c291c5-dacc-406a-b637-7c86940848f1-kube-api-access-m8s9v\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:11:05.487779 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.487742 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632ec6a-1247-47cd-b436-8c3281b21eda-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:11:05.487779 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.487751 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hrm\" (UniqueName: \"kubernetes.io/projected/0632ec6a-1247-47cd-b436-8c3281b21eda-kube-api-access-z6hrm\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:11:05.632037 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.632001 2573 generic.go:356] "Generic (PLEG): container finished" podID="34c291c5-dacc-406a-b637-7c86940848f1" containerID="06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149" exitCode=0 Jul 27 20:11:05.632234 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.632051 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" Jul 27 20:11:05.632234 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.632089 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" event={"ID":"34c291c5-dacc-406a-b637-7c86940848f1","Type":"ContainerDied","Data":"06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149"} Jul 27 20:11:05.632234 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.632152 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b" event={"ID":"34c291c5-dacc-406a-b637-7c86940848f1","Type":"ContainerDied","Data":"55bc330609b240aa7b8ea223d9569550ba7e3c68e228fa154468a75d600be4b9"} Jul 27 20:11:05.632234 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.632184 2573 scope.go:117] "RemoveContainer" containerID="06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149" Jul 27 20:11:05.633572 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.633543 2573 generic.go:356] "Generic (PLEG): container finished" podID="0632ec6a-1247-47cd-b436-8c3281b21eda" containerID="4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e" exitCode=0 Jul 27 20:11:05.633669 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.633572 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" event={"ID":"0632ec6a-1247-47cd-b436-8c3281b21eda","Type":"ContainerDied","Data":"4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e"} Jul 27 20:11:05.633669 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.633605 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" Jul 27 20:11:05.633777 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.633611 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd" event={"ID":"0632ec6a-1247-47cd-b436-8c3281b21eda","Type":"ContainerDied","Data":"eeb5639891fdd4ce8001179f6f1abda23fb62e5aea815be381f920b2b1ecf44d"} Jul 27 20:11:05.642180 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.642153 2573 scope.go:117] "RemoveContainer" containerID="06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149" Jul 27 20:11:05.642602 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:11:05.642480 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149\": container with ID starting with 06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149 not found: ID does not exist" containerID="06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149" Jul 27 20:11:05.642602 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.642510 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149"} err="failed to get container status \"06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149\": rpc error: code = NotFound desc = could not find container \"06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149\": container with ID starting with 06824621bfb620a40d13af986030510f479243a8fc085e599e98f11c21e32149 not found: ID does not exist" Jul 27 20:11:05.642602 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.642526 2573 scope.go:117] "RemoveContainer" containerID="4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e" Jul 27 20:11:05.651019 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.650999 2573 scope.go:117] "RemoveContainer" containerID="4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e" Jul 27 20:11:05.651265 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:11:05.651247 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e\": container with ID starting with 4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e not found: ID does not exist" containerID="4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e" Jul 27 20:11:05.651366 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.651272 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e"} err="failed to get container status \"4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e\": rpc error: code = NotFound desc = could not find container \"4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e\": container with ID starting with 4ac9f336cfde6a61c1b736f14a5d185cbf95467e068392d8c84684286509c70e not found: ID does not exist" Jul 27 20:11:05.658404 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.658380 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd"] Jul 27 20:11:05.662057 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.662034 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-452dddb4-b76f9cf4d-8vcmd"] Jul 27 20:11:05.674632 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.674608 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b"] Jul 27 20:11:05.678124 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.678102 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-452dddb4-7c95f787db-xp29b"] Jul 27 20:11:05.875860 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.875779 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0632ec6a-1247-47cd-b436-8c3281b21eda" path="/var/lib/kubelet/pods/0632ec6a-1247-47cd-b436-8c3281b21eda/volumes" Jul 27 20:11:05.876133 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:05.876120 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34c291c5-dacc-406a-b637-7c86940848f1" path="/var/lib/kubelet/pods/34c291c5-dacc-406a-b637-7c86940848f1/volumes" Jul 27 20:11:08.033181 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:08.033146 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:11:09.720725 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:09.720693 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:09.725631 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:09.725599 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf"] Jul 27 20:11:09.728177 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:09.728156 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-452dddb4-rwbzf"] Jul 27 20:11:09.875172 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:09.875139 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b813fb9-5b95-45c7-906f-0e74a8b82ac4" path="/var/lib/kubelet/pods/7b813fb9-5b95-45c7-906f-0e74a8b82ac4/volumes" Jul 27 20:11:10.068106 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:10.068021 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:15.994279 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:15.994193 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:16.095976 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:16.095934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:16.280478 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:16.280387 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:16.500978 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:16.500943 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:16.637230 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:16.637134 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:16.749087 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:16.749049 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:16.849571 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:16.849541 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:23.105950 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:23.105915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:23.224544 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:23.224509 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:23.402042 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:23.401951 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:23.506807 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:23.506772 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:29.343159 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:29.343123 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:29.446331 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:29.446277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:36.142091 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:36.142056 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:36.461276 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:36.461244 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:37.158572 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:37.158538 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:39.643066 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:39.643030 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:39.739962 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:39.739929 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:39.870436 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:39.870408 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:40.125149 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.125119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:40.464542 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.464510 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:40.559242 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.559202 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:40.584584 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.584550 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj"] Jul 27 20:11:40.584957 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.584944 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0632ec6a-1247-47cd-b436-8c3281b21eda" containerName="payload-pre-processing" Jul 27 20:11:40.585011 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.584960 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0632ec6a-1247-47cd-b436-8c3281b21eda" containerName="payload-pre-processing" Jul 27 20:11:40.585011 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.584970 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c291c5-dacc-406a-b637-7c86940848f1" containerName="payload-processing" Jul 27 20:11:40.585011 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.584976 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c291c5-dacc-406a-b637-7c86940848f1" containerName="payload-processing" Jul 27 20:11:40.585011 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.584985 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b813fb9-5b95-45c7-906f-0e74a8b82ac4" containerName="revoke-keys" Jul 27 20:11:40.585011 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.584990 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b813fb9-5b95-45c7-906f-0e74a8b82ac4" containerName="revoke-keys" Jul 27 20:11:40.585181 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.585047 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0632ec6a-1247-47cd-b436-8c3281b21eda" containerName="payload-pre-processing" Jul 27 20:11:40.585181 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.585068 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b813fb9-5b95-45c7-906f-0e74a8b82ac4" containerName="revoke-keys" Jul 27 20:11:40.585181 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.585074 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c291c5-dacc-406a-b637-7c86940848f1" containerName="payload-processing" Jul 27 20:11:40.588220 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.588204 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:40.590893 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.590874 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-25d7f4d7\"" Jul 27 20:11:40.591020 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.590878 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-25d7f4d7-dockercfg-4kcg2\"" Jul 27 20:11:40.597916 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.597893 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj"] Jul 27 20:11:40.637425 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.637387 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p"] Jul 27 20:11:40.640866 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.640851 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:40.650930 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.650873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p"] Jul 27 20:11:40.660756 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.660729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b97c\" (UniqueName: \"kubernetes.io/projected/0222c5a0-3921-4f50-8585-8685cfdaed83-kube-api-access-4b97c\") pod \"payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj\" (UID: \"0222c5a0-3921-4f50-8585-8685cfdaed83\") " pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:40.660875 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.660786 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0222c5a0-3921-4f50-8585-8685cfdaed83-plugins-config-volume\") pod \"payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj\" (UID: \"0222c5a0-3921-4f50-8585-8685cfdaed83\") " pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:40.762328 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.762219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b97c\" (UniqueName: \"kubernetes.io/projected/0222c5a0-3921-4f50-8585-8685cfdaed83-kube-api-access-4b97c\") pod \"payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj\" (UID: \"0222c5a0-3921-4f50-8585-8685cfdaed83\") " pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:40.762328 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.762259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db8c5\" (UniqueName: \"kubernetes.io/projected/b9a4d135-5259-4d5c-9199-f8696c1ca59f-kube-api-access-db8c5\") pod \"payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p\" (UID: \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:40.762328 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.762328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0222c5a0-3921-4f50-8585-8685cfdaed83-plugins-config-volume\") pod \"payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj\" (UID: \"0222c5a0-3921-4f50-8585-8685cfdaed83\") " pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:40.762568 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.762430 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4d135-5259-4d5c-9199-f8696c1ca59f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p\" (UID: \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:40.762985 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.762964 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0222c5a0-3921-4f50-8585-8685cfdaed83-plugins-config-volume\") pod \"payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj\" (UID: \"0222c5a0-3921-4f50-8585-8685cfdaed83\") " pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:40.774460 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.774435 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b97c\" (UniqueName: \"kubernetes.io/projected/0222c5a0-3921-4f50-8585-8685cfdaed83-kube-api-access-4b97c\") pod \"payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj\" (UID: \"0222c5a0-3921-4f50-8585-8685cfdaed83\") " pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:40.863817 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.863783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4d135-5259-4d5c-9199-f8696c1ca59f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p\" (UID: \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:40.863989 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.863840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db8c5\" (UniqueName: \"kubernetes.io/projected/b9a4d135-5259-4d5c-9199-f8696c1ca59f-kube-api-access-db8c5\") pod \"payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p\" (UID: \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:40.864491 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.864473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4d135-5259-4d5c-9199-f8696c1ca59f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p\" (UID: \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:40.877374 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.877349 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db8c5\" (UniqueName: \"kubernetes.io/projected/b9a4d135-5259-4d5c-9199-f8696c1ca59f-kube-api-access-db8c5\") pod \"payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p\" (UID: \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:40.899768 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.899739 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:40.951753 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:40.951722 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:41.042844 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.041428 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj"] Jul 27 20:11:41.043734 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:11:41.043426 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0222c5a0_3921_4f50_8585_8685cfdaed83.slice/crio-78ac94e1ea2383a5d6de1f350361bfa015f96f80fcb4683acaf59fcc7ce4102e WatchSource:0}: Error finding container 78ac94e1ea2383a5d6de1f350361bfa015f96f80fcb4683acaf59fcc7ce4102e: Status 404 returned error can't find the container with id 78ac94e1ea2383a5d6de1f350361bfa015f96f80fcb4683acaf59fcc7ce4102e Jul 27 20:11:41.120914 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.120886 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p"] Jul 27 20:11:41.122235 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:11:41.122211 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a4d135_5259_4d5c_9199_f8696c1ca59f.slice/crio-ee71896978a626166ede23859e91d379b8b6581329380c9113970edb48f41828 WatchSource:0}: Error finding container ee71896978a626166ede23859e91d379b8b6581329380c9113970edb48f41828: Status 404 returned error can't find the container with id ee71896978a626166ede23859e91d379b8b6581329380c9113970edb48f41828 Jul 27 20:11:41.839385 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.839341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" event={"ID":"0222c5a0-3921-4f50-8585-8685cfdaed83","Type":"ContainerStarted","Data":"c28e87788ea5dbc9e3f95c74551ca6e569aff2a8c7e081498ccbfa73cbc8f7ca"} Jul 27 20:11:41.839863 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.839405 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" event={"ID":"0222c5a0-3921-4f50-8585-8685cfdaed83","Type":"ContainerStarted","Data":"78ac94e1ea2383a5d6de1f350361bfa015f96f80fcb4683acaf59fcc7ce4102e"} Jul 27 20:11:41.839863 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.839433 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:41.840878 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.840848 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" event={"ID":"b9a4d135-5259-4d5c-9199-f8696c1ca59f","Type":"ContainerStarted","Data":"9c13f246885461c8bae3fcdf97d2b65f9bbb42998a0fcebe7ca2b9e87be8bfe1"} Jul 27 20:11:41.840990 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.840891 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" event={"ID":"b9a4d135-5259-4d5c-9199-f8696c1ca59f","Type":"ContainerStarted","Data":"ee71896978a626166ede23859e91d379b8b6581329380c9113970edb48f41828"} Jul 27 20:11:41.840990 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.840922 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:41.858367 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.858316 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" podStartSLOduration=1.858286834 podStartE2EDuration="1.858286834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:41.855630203 +0000 UTC m=+1906.468214798" watchObservedRunningTime="2026-07-27 20:11:41.858286834 +0000 UTC m=+1906.470871417" Jul 27 20:11:41.875524 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:41.875476 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" podStartSLOduration=1.875460481 podStartE2EDuration="1.875460481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:41.875218777 +0000 UTC m=+1906.487803356" watchObservedRunningTime="2026-07-27 20:11:41.875460481 +0000 UTC m=+1906.488045065" Jul 27 20:11:44.114877 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:44.114845 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:11:46.235364 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:46.235324 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:11:52.848681 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:52.848639 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:11:52.849080 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:52.848713 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:11:59.784716 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:11:59.784683 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:12:03.139060 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:03.139020 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:12:11.084515 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:11.084477 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:12:34.355928 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:34.355888 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:12:39.074603 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.074568 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78"] Jul 27 20:12:39.078076 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.078059 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" Jul 27 20:12:39.080752 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.080732 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:12:39.081044 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.081030 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:12:39.089762 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.089725 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78"] Jul 27 20:12:39.162594 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.162554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0812975-9c1b-4dc9-b63e-0018b8800e50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78\" (UID: \"a0812975-9c1b-4dc9-b63e-0018b8800e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" Jul 27 20:12:39.263639 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.263600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0812975-9c1b-4dc9-b63e-0018b8800e50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78\" (UID: \"a0812975-9c1b-4dc9-b63e-0018b8800e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" Jul 27 20:12:39.264266 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.264240 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0812975-9c1b-4dc9-b63e-0018b8800e50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78\" (UID: \"a0812975-9c1b-4dc9-b63e-0018b8800e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" Jul 27 20:12:39.388985 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.388903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" Jul 27 20:12:39.523466 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:39.523439 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78"] Jul 27 20:12:39.525248 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:12:39.525223 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0812975_9c1b_4dc9_b63e_0018b8800e50.slice/crio-42d69d70043a67c47a25eb1c67f05c7e08fbe67367dae57c055884b729397bf9 WatchSource:0}: Error finding container 42d69d70043a67c47a25eb1c67f05c7e08fbe67367dae57c055884b729397bf9: Status 404 returned error can't find the container with id 42d69d70043a67c47a25eb1c67f05c7e08fbe67367dae57c055884b729397bf9 Jul 27 20:12:40.156710 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:40.156612 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" event={"ID":"a0812975-9c1b-4dc9-b63e-0018b8800e50","Type":"ContainerStarted","Data":"3a98a147204a084c734f29505029700551d3d204e402e30321a435bf9789f250"} Jul 27 20:12:40.156710 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:40.156671 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" event={"ID":"a0812975-9c1b-4dc9-b63e-0018b8800e50","Type":"ContainerStarted","Data":"42d69d70043a67c47a25eb1c67f05c7e08fbe67367dae57c055884b729397bf9"} Jul 27 20:12:40.178474 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:40.178425 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" podStartSLOduration=1.178408976 podStartE2EDuration="1.178408976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:12:40.175274827 +0000 UTC m=+1964.787859410" watchObservedRunningTime="2026-07-27 20:12:40.178408976 +0000 UTC m=+1964.790993558" Jul 27 20:12:41.163403 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:41.163365 2573 generic.go:356] "Generic (PLEG): container finished" podID="a0812975-9c1b-4dc9-b63e-0018b8800e50" containerID="3a98a147204a084c734f29505029700551d3d204e402e30321a435bf9789f250" exitCode=0 Jul 27 20:12:41.163832 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:41.163453 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" event={"ID":"a0812975-9c1b-4dc9-b63e-0018b8800e50","Type":"ContainerDied","Data":"3a98a147204a084c734f29505029700551d3d204e402e30321a435bf9789f250"} Jul 27 20:12:42.302348 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:42.302325 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" Jul 27 20:12:42.390001 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:42.389968 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0812975-9c1b-4dc9-b63e-0018b8800e50-maas-api-service-ca\") pod \"a0812975-9c1b-4dc9-b63e-0018b8800e50\" (UID: \"a0812975-9c1b-4dc9-b63e-0018b8800e50\") " Jul 27 20:12:42.390336 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:42.390314 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0812975-9c1b-4dc9-b63e-0018b8800e50-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0812975-9c1b-4dc9-b63e-0018b8800e50" (UID: "a0812975-9c1b-4dc9-b63e-0018b8800e50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:12:42.490708 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:42.490678 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0812975-9c1b-4dc9-b63e-0018b8800e50-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:12:43.176002 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:43.175968 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" Jul 27 20:12:43.176002 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:43.175979 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78" event={"ID":"a0812975-9c1b-4dc9-b63e-0018b8800e50","Type":"ContainerDied","Data":"42d69d70043a67c47a25eb1c67f05c7e08fbe67367dae57c055884b729397bf9"} Jul 27 20:12:43.176213 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:43.176023 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d69d70043a67c47a25eb1c67f05c7e08fbe67367dae57c055884b729397bf9" Jul 27 20:12:53.552371 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:53.552333 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:12:54.934022 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:54.933984 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:12:54.983387 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:54.983357 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj"] Jul 27 20:12:54.983600 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:54.983564 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" podUID="0222c5a0-3921-4f50-8585-8685cfdaed83" containerName="payload-processing" containerID="cri-o://c28e87788ea5dbc9e3f95c74551ca6e569aff2a8c7e081498ccbfa73cbc8f7ca" gracePeriod=30 Jul 27 20:12:55.000272 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.000247 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p"] Jul 27 20:12:55.000490 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.000462 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" podUID="b9a4d135-5259-4d5c-9199-f8696c1ca59f" containerName="payload-pre-processing" containerID="cri-o://9c13f246885461c8bae3fcdf97d2b65f9bbb42998a0fcebe7ca2b9e87be8bfe1" gracePeriod=30 Jul 27 20:12:55.243997 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.243961 2573 generic.go:356] "Generic (PLEG): container finished" podID="0222c5a0-3921-4f50-8585-8685cfdaed83" containerID="c28e87788ea5dbc9e3f95c74551ca6e569aff2a8c7e081498ccbfa73cbc8f7ca" exitCode=0 Jul 27 20:12:55.244155 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.244026 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" event={"ID":"0222c5a0-3921-4f50-8585-8685cfdaed83","Type":"ContainerDied","Data":"c28e87788ea5dbc9e3f95c74551ca6e569aff2a8c7e081498ccbfa73cbc8f7ca"} Jul 27 20:12:55.245980 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.245952 2573 generic.go:356] "Generic (PLEG): container finished" podID="b9a4d135-5259-4d5c-9199-f8696c1ca59f" containerID="9c13f246885461c8bae3fcdf97d2b65f9bbb42998a0fcebe7ca2b9e87be8bfe1" exitCode=0 Jul 27 20:12:55.246110 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.246022 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" event={"ID":"b9a4d135-5259-4d5c-9199-f8696c1ca59f","Type":"ContainerDied","Data":"9c13f246885461c8bae3fcdf97d2b65f9bbb42998a0fcebe7ca2b9e87be8bfe1"} Jul 27 20:12:55.274703 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.274683 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:12:55.277603 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.277582 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:12:55.414057 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.414025 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4d135-5259-4d5c-9199-f8696c1ca59f-plugins-config-volume\") pod \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\" (UID: \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\") " Jul 27 20:12:55.414222 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.414126 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db8c5\" (UniqueName: \"kubernetes.io/projected/b9a4d135-5259-4d5c-9199-f8696c1ca59f-kube-api-access-db8c5\") pod \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\" (UID: \"b9a4d135-5259-4d5c-9199-f8696c1ca59f\") " Jul 27 20:12:55.414222 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.414182 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b97c\" (UniqueName: \"kubernetes.io/projected/0222c5a0-3921-4f50-8585-8685cfdaed83-kube-api-access-4b97c\") pod \"0222c5a0-3921-4f50-8585-8685cfdaed83\" (UID: \"0222c5a0-3921-4f50-8585-8685cfdaed83\") " Jul 27 20:12:55.414222 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.414209 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0222c5a0-3921-4f50-8585-8685cfdaed83-plugins-config-volume\") pod \"0222c5a0-3921-4f50-8585-8685cfdaed83\" (UID: \"0222c5a0-3921-4f50-8585-8685cfdaed83\") " Jul 27 20:12:55.414556 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.414530 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a4d135-5259-4d5c-9199-f8696c1ca59f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9a4d135-5259-4d5c-9199-f8696c1ca59f" (UID: "b9a4d135-5259-4d5c-9199-f8696c1ca59f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:12:55.414662 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.414641 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0222c5a0-3921-4f50-8585-8685cfdaed83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0222c5a0-3921-4f50-8585-8685cfdaed83" (UID: "0222c5a0-3921-4f50-8585-8685cfdaed83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:12:55.416379 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.416360 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0222c5a0-3921-4f50-8585-8685cfdaed83-kube-api-access-4b97c" (OuterVolumeSpecName: "kube-api-access-4b97c") pod "0222c5a0-3921-4f50-8585-8685cfdaed83" (UID: "0222c5a0-3921-4f50-8585-8685cfdaed83"). InnerVolumeSpecName "kube-api-access-4b97c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:12:55.416446 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.416395 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a4d135-5259-4d5c-9199-f8696c1ca59f-kube-api-access-db8c5" (OuterVolumeSpecName: "kube-api-access-db8c5") pod "b9a4d135-5259-4d5c-9199-f8696c1ca59f" (UID: "b9a4d135-5259-4d5c-9199-f8696c1ca59f"). InnerVolumeSpecName "kube-api-access-db8c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:12:55.515114 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.515029 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4d135-5259-4d5c-9199-f8696c1ca59f-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:12:55.515114 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.515056 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db8c5\" (UniqueName: \"kubernetes.io/projected/b9a4d135-5259-4d5c-9199-f8696c1ca59f-kube-api-access-db8c5\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:12:55.515114 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.515067 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b97c\" (UniqueName: \"kubernetes.io/projected/0222c5a0-3921-4f50-8585-8685cfdaed83-kube-api-access-4b97c\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:12:55.515114 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:55.515076 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0222c5a0-3921-4f50-8585-8685cfdaed83-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:12:56.035791 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.035760 2573 scope.go:117] "RemoveContainer" containerID="c74b66fd22da119a2f8992ead2c69a8694720a5bf4f1b8199f309b12849e8a8e" Jul 27 20:12:56.215733 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.215698 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:12:56.253935 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.253906 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" Jul 27 20:12:56.254106 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.253954 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p" event={"ID":"b9a4d135-5259-4d5c-9199-f8696c1ca59f","Type":"ContainerDied","Data":"ee71896978a626166ede23859e91d379b8b6581329380c9113970edb48f41828"} Jul 27 20:12:56.254106 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.254012 2573 scope.go:117] "RemoveContainer" containerID="9c13f246885461c8bae3fcdf97d2b65f9bbb42998a0fcebe7ca2b9e87be8bfe1" Jul 27 20:12:56.255483 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.255454 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" Jul 27 20:12:56.255575 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.255461 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj" event={"ID":"0222c5a0-3921-4f50-8585-8685cfdaed83","Type":"ContainerDied","Data":"78ac94e1ea2383a5d6de1f350361bfa015f96f80fcb4683acaf59fcc7ce4102e"} Jul 27 20:12:56.263085 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.263065 2573 scope.go:117] "RemoveContainer" containerID="c28e87788ea5dbc9e3f95c74551ca6e569aff2a8c7e081498ccbfa73cbc8f7ca" Jul 27 20:12:56.275681 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.275652 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj"] Jul 27 20:12:56.280472 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.280440 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25d7f4d7-6ff858f854-6s2vj"] Jul 27 20:12:56.295044 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.295010 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p"] Jul 27 20:12:56.302138 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:56.302108 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25d7f4d7-5f5d474784-jp96p"] Jul 27 20:12:57.874988 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:57.874956 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0222c5a0-3921-4f50-8585-8685cfdaed83" path="/var/lib/kubelet/pods/0222c5a0-3921-4f50-8585-8685cfdaed83/volumes" Jul 27 20:12:57.875364 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:57.875320 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a4d135-5259-4d5c-9199-f8696c1ca59f" path="/var/lib/kubelet/pods/b9a4d135-5259-4d5c-9199-f8696c1ca59f/volumes" Jul 27 20:12:59.247558 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:59.247520 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78"] Jul 27 20:12:59.251848 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:59.251815 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:12:59.251996 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:59.251826 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25d7f4d7-6gc78"] Jul 27 20:12:59.875861 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:12:59.875819 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0812975-9c1b-4dc9-b63e-0018b8800e50" path="/var/lib/kubelet/pods/a0812975-9c1b-4dc9-b63e-0018b8800e50/volumes" Jul 27 20:13:02.384839 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:02.384805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:02.446697 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:02.446660 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:13:07.470974 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:07.470942 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:43.692419 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:43.692378 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:44.850067 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:44.850029 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:50.523957 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:50.523923 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:50.603640 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:50.603603 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:50.795420 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:50.795336 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:51.064534 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:51.064433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:52.102243 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:52.102191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:52.199878 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:52.199837 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:52.301059 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:52.301026 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:52.970689 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:52.970637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:53.076607 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:53.076573 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:53.171909 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:53.171878 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:54.498433 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:54.498396 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:13:54.940554 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:54.940523 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:13:56.057803 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:13:56.057774 2573 scope.go:117] "RemoveContainer" containerID="e513054ad55f958e2175be75eee5d508e500cd1c5f2880896d7af5c9bbefbf93" Jul 27 20:14:00.496065 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:00.496023 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:00.562690 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:00.562658 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:01.168813 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:01.168780 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:06.765922 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:06.765890 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:14:07.595802 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:07.595760 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:07.710215 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:07.710179 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:07.798825 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:07.798782 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:18.301383 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:18.301273 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:18.655993 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:18.655903 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:22.219765 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:22.219723 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:22.278074 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:22.278038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:39.596789 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:39.596750 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:41.692838 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:41.692800 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:46.230777 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:46.230740 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:46.481723 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:46.481627 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:46.692347 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:46.692308 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:47.035653 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:47.035615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:47.829357 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:47.829317 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:49.712815 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:49.712773 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:49.813742 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:49.813630 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:49.912812 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:49.912779 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:50.145652 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.145561 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:50.446217 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.446183 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:50.545986 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.545944 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:50.568871 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.568842 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66"] Jul 27 20:14:50.569216 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.569204 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a4d135-5259-4d5c-9199-f8696c1ca59f" containerName="payload-pre-processing" Jul 27 20:14:50.569265 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.569219 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a4d135-5259-4d5c-9199-f8696c1ca59f" containerName="payload-pre-processing" Jul 27 20:14:50.569265 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.569230 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0812975-9c1b-4dc9-b63e-0018b8800e50" containerName="revoke-keys" Jul 27 20:14:50.569265 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.569237 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0812975-9c1b-4dc9-b63e-0018b8800e50" containerName="revoke-keys" Jul 27 20:14:50.569265 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.569248 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0222c5a0-3921-4f50-8585-8685cfdaed83" containerName="payload-processing" Jul 27 20:14:50.569265 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.569253 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0222c5a0-3921-4f50-8585-8685cfdaed83" containerName="payload-processing" Jul 27 20:14:50.569446 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.569319 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0812975-9c1b-4dc9-b63e-0018b8800e50" containerName="revoke-keys" Jul 27 20:14:50.569446 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.569329 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a4d135-5259-4d5c-9199-f8696c1ca59f" containerName="payload-pre-processing" Jul 27 20:14:50.569446 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.569338 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0222c5a0-3921-4f50-8585-8685cfdaed83" containerName="payload-processing" Jul 27 20:14:50.572256 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.572240 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:14:50.575971 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.575940 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-849f6b\"" Jul 27 20:14:50.584883 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.584859 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66"] Jul 27 20:14:50.592161 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.592138 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq"] Jul 27 20:14:50.595580 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.595561 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:14:50.598123 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.598106 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-849f6b-dockercfg-vvkdd\"" Jul 27 20:14:50.607016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.606989 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq"] Jul 27 20:14:50.717431 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.717348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a35a7-4637-4320-9c25-92a092822e29-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq\" (UID: \"1e8a35a7-4637-4320-9c25-92a092822e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:14:50.717431 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.717385 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffd584f-4bd7-4b00-8c48-c7e85c827646-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66\" (UID: \"dffd584f-4bd7-4b00-8c48-c7e85c827646\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:14:50.717826 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.717476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpj7v\" (UniqueName: \"kubernetes.io/projected/dffd584f-4bd7-4b00-8c48-c7e85c827646-kube-api-access-qpj7v\") pod \"payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66\" (UID: \"dffd584f-4bd7-4b00-8c48-c7e85c827646\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:14:50.717826 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.717554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44swx\" (UniqueName: \"kubernetes.io/projected/1e8a35a7-4637-4320-9c25-92a092822e29-kube-api-access-44swx\") pod \"payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq\" (UID: \"1e8a35a7-4637-4320-9c25-92a092822e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:14:50.818768 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.818731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44swx\" (UniqueName: \"kubernetes.io/projected/1e8a35a7-4637-4320-9c25-92a092822e29-kube-api-access-44swx\") pod \"payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq\" (UID: \"1e8a35a7-4637-4320-9c25-92a092822e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:14:50.818950 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.818788 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a35a7-4637-4320-9c25-92a092822e29-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq\" (UID: \"1e8a35a7-4637-4320-9c25-92a092822e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:14:50.818950 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.818820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffd584f-4bd7-4b00-8c48-c7e85c827646-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66\" (UID: \"dffd584f-4bd7-4b00-8c48-c7e85c827646\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:14:50.818950 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.818860 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpj7v\" (UniqueName: \"kubernetes.io/projected/dffd584f-4bd7-4b00-8c48-c7e85c827646-kube-api-access-qpj7v\") pod \"payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66\" (UID: \"dffd584f-4bd7-4b00-8c48-c7e85c827646\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:14:50.819624 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.819605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a35a7-4637-4320-9c25-92a092822e29-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq\" (UID: \"1e8a35a7-4637-4320-9c25-92a092822e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:14:50.819624 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.819614 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffd584f-4bd7-4b00-8c48-c7e85c827646-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66\" (UID: \"dffd584f-4bd7-4b00-8c48-c7e85c827646\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:14:50.828635 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.828604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpj7v\" (UniqueName: \"kubernetes.io/projected/dffd584f-4bd7-4b00-8c48-c7e85c827646-kube-api-access-qpj7v\") pod \"payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66\" (UID: \"dffd584f-4bd7-4b00-8c48-c7e85c827646\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:14:50.828769 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.828708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44swx\" (UniqueName: \"kubernetes.io/projected/1e8a35a7-4637-4320-9c25-92a092822e29-kube-api-access-44swx\") pod \"payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq\" (UID: \"1e8a35a7-4637-4320-9c25-92a092822e29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:14:50.883649 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.883610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:14:50.909888 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:50.909860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:14:51.036710 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.036680 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66"] Jul 27 20:14:51.038445 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:14:51.038410 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffd584f_4bd7_4b00_8c48_c7e85c827646.slice/crio-00e3a425e9925058a9c85236118af5ed6c065478f3d32ea7ae7d0f95e5bbcc6e WatchSource:0}: Error finding container 00e3a425e9925058a9c85236118af5ed6c065478f3d32ea7ae7d0f95e5bbcc6e: Status 404 returned error can't find the container with id 00e3a425e9925058a9c85236118af5ed6c065478f3d32ea7ae7d0f95e5bbcc6e Jul 27 20:14:51.064803 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.064781 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq"] Jul 27 20:14:51.072466 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:14:51.072441 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8a35a7_4637_4320_9c25_92a092822e29.slice/crio-47ac0e8ffb5a8bd2e256427f251688c84c6ab0bd4e049e31e2b62866e51b626f WatchSource:0}: Error finding container 47ac0e8ffb5a8bd2e256427f251688c84c6ab0bd4e049e31e2b62866e51b626f: Status 404 returned error can't find the container with id 47ac0e8ffb5a8bd2e256427f251688c84c6ab0bd4e049e31e2b62866e51b626f Jul 27 20:14:51.875015 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.874981 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:14:51.875479 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.875039 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" event={"ID":"dffd584f-4bd7-4b00-8c48-c7e85c827646","Type":"ContainerStarted","Data":"1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca"} Jul 27 20:14:51.875479 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.875065 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" event={"ID":"dffd584f-4bd7-4b00-8c48-c7e85c827646","Type":"ContainerStarted","Data":"00e3a425e9925058a9c85236118af5ed6c065478f3d32ea7ae7d0f95e5bbcc6e"} Jul 27 20:14:51.876087 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.876067 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" event={"ID":"1e8a35a7-4637-4320-9c25-92a092822e29","Type":"ContainerStarted","Data":"93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8"} Jul 27 20:14:51.876153 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.876105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" event={"ID":"1e8a35a7-4637-4320-9c25-92a092822e29","Type":"ContainerStarted","Data":"47ac0e8ffb5a8bd2e256427f251688c84c6ab0bd4e049e31e2b62866e51b626f"} Jul 27 20:14:51.876188 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.876159 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:14:51.896553 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.896499 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" podStartSLOduration=1.896478891 podStartE2EDuration="1.896478891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:51.89554913 +0000 UTC m=+2096.508133729" watchObservedRunningTime="2026-07-27 20:14:51.896478891 +0000 UTC m=+2096.509063477" Jul 27 20:14:51.918447 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:51.918391 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" podStartSLOduration=1.918373739 podStartE2EDuration="1.918373739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:51.916750275 +0000 UTC m=+2096.529334859" watchObservedRunningTime="2026-07-27 20:14:51.918373739 +0000 UTC m=+2096.530958321" Jul 27 20:14:54.178238 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:54.178195 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:54.522257 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:54.522163 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:54.835387 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:54.835279 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:55.424844 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:55.424809 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:56.089422 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:56.089385 2573 scope.go:117] "RemoveContainer" containerID="1e4b55bf18d963775e10a3ed5311b2de60176dc040acd80be62fca98fdf46b0f" Jul 27 20:14:56.098077 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:56.098047 2573 scope.go:117] "RemoveContainer" containerID="2da3e1a0ab03bb963e8dd1c89d6c8caf30b8323c3e0f294b676f923a445c4638" Jul 27 20:14:57.544337 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:57.544285 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:14:57.776284 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:57.776253 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:57.877981 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:57.877890 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:57.950499 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:57.950457 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:58.263011 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.262973 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:58.525763 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.525651 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:58.621159 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.621121 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:14:58.662073 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.662038 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c"] Jul 27 20:14:58.665543 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.665528 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:14:58.668228 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.668209 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0a5392\"" Jul 27 20:14:58.668228 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.668222 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0a5392-dockercfg-47lpc\"" Jul 27 20:14:58.678197 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.678175 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c"] Jul 27 20:14:58.719445 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.719410 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm"] Jul 27 20:14:58.722817 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.722793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:14:58.735013 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.734985 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm"] Jul 27 20:14:58.791998 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.791921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85jk\" (UniqueName: \"kubernetes.io/projected/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-kube-api-access-s85jk\") pod \"payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c\" (UID: \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:14:58.791998 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.791974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c\" (UID: \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:14:58.892603 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.892563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c\" (UID: \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:14:58.892603 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.892607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lfw\" (UniqueName: \"kubernetes.io/projected/e54198f9-fc95-4003-abbf-a2413e25e331-kube-api-access-c5lfw\") pod \"payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm\" (UID: \"e54198f9-fc95-4003-abbf-a2413e25e331\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:14:58.892889 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.892680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54198f9-fc95-4003-abbf-a2413e25e331-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm\" (UID: \"e54198f9-fc95-4003-abbf-a2413e25e331\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:14:58.892889 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.892705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85jk\" (UniqueName: \"kubernetes.io/projected/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-kube-api-access-s85jk\") pod \"payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c\" (UID: \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:14:58.893509 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.893484 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c\" (UID: \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:14:58.901712 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.901682 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85jk\" (UniqueName: \"kubernetes.io/projected/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-kube-api-access-s85jk\") pod \"payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c\" (UID: \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:14:58.975470 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.975429 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:14:58.993825 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.993790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54198f9-fc95-4003-abbf-a2413e25e331-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm\" (UID: \"e54198f9-fc95-4003-abbf-a2413e25e331\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:14:58.993972 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.993924 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lfw\" (UniqueName: \"kubernetes.io/projected/e54198f9-fc95-4003-abbf-a2413e25e331-kube-api-access-c5lfw\") pod \"payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm\" (UID: \"e54198f9-fc95-4003-abbf-a2413e25e331\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:14:58.994737 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:58.994711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54198f9-fc95-4003-abbf-a2413e25e331-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm\" (UID: \"e54198f9-fc95-4003-abbf-a2413e25e331\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:14:59.006408 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.006378 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lfw\" (UniqueName: \"kubernetes.io/projected/e54198f9-fc95-4003-abbf-a2413e25e331-kube-api-access-c5lfw\") pod \"payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm\" (UID: \"e54198f9-fc95-4003-abbf-a2413e25e331\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:14:59.034170 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.034137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:14:59.122308 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.122267 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c"] Jul 27 20:14:59.124289 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:14:59.124263 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698cd6d7_52a5_4ee9_8a59_c9b0dc46c5d7.slice/crio-051ecf436e074adaf865b9c0dc8b69aa2cb46b02f6f1fb8f8e3599aea5458d6c WatchSource:0}: Error finding container 051ecf436e074adaf865b9c0dc8b69aa2cb46b02f6f1fb8f8e3599aea5458d6c: Status 404 returned error can't find the container with id 051ecf436e074adaf865b9c0dc8b69aa2cb46b02f6f1fb8f8e3599aea5458d6c Jul 27 20:14:59.182140 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.182118 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm"] Jul 27 20:14:59.184442 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:14:59.184415 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54198f9_fc95_4003_abbf_a2413e25e331.slice/crio-83be414bc8e8e8e4306dbd87f4e8c574c12316c9461f6d04d47644f4d3fcd7a1 WatchSource:0}: Error finding container 83be414bc8e8e8e4306dbd87f4e8c574c12316c9461f6d04d47644f4d3fcd7a1: Status 404 returned error can't find the container with id 83be414bc8e8e8e4306dbd87f4e8c574c12316c9461f6d04d47644f4d3fcd7a1 Jul 27 20:14:59.922289 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.922251 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" event={"ID":"e54198f9-fc95-4003-abbf-a2413e25e331","Type":"ContainerStarted","Data":"d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b"} Jul 27 20:14:59.922786 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.922341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" event={"ID":"e54198f9-fc95-4003-abbf-a2413e25e331","Type":"ContainerStarted","Data":"83be414bc8e8e8e4306dbd87f4e8c574c12316c9461f6d04d47644f4d3fcd7a1"} Jul 27 20:14:59.922786 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.922375 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:14:59.923859 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.923834 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" event={"ID":"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7","Type":"ContainerStarted","Data":"7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d"} Jul 27 20:14:59.923997 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.923886 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:14:59.923997 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.923909 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" event={"ID":"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7","Type":"ContainerStarted","Data":"051ecf436e074adaf865b9c0dc8b69aa2cb46b02f6f1fb8f8e3599aea5458d6c"} Jul 27 20:14:59.939112 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.938970 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" podStartSLOduration=1.938953611 podStartE2EDuration="1.938953611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:59.938576223 +0000 UTC m=+2104.551160806" watchObservedRunningTime="2026-07-27 20:14:59.938953611 +0000 UTC m=+2104.551538193" Jul 27 20:14:59.955208 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:14:59.955160 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" podStartSLOduration=1.955146853 podStartE2EDuration="1.955146853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:59.95415216 +0000 UTC m=+2104.566736746" watchObservedRunningTime="2026-07-27 20:14:59.955146853 +0000 UTC m=+2104.567731436" Jul 27 20:15:00.133075 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.133044 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf"] Jul 27 20:15:00.136806 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.136784 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8"] Jul 27 20:15:00.136945 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.136912 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" Jul 27 20:15:00.139468 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.139450 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:15:00.140236 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.140218 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8" Jul 27 20:15:00.146470 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.146448 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf"] Jul 27 20:15:00.150142 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.150123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" Jul 27 20:15:00.157667 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.157644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8" Jul 27 20:15:00.161962 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.161934 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8"] Jul 27 20:15:00.311261 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.311229 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf"] Jul 27 20:15:00.315320 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:15:00.315266 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e74419_6790_4a83_a3ff_c4cbd0f4465d.slice/crio-d6265f06a8a67652fc3561bf0b41ca2fd365157a0c6952bcc5a9d38b92cffc16 WatchSource:0}: Error finding container d6265f06a8a67652fc3561bf0b41ca2fd365157a0c6952bcc5a9d38b92cffc16: Status 404 returned error can't find the container with id d6265f06a8a67652fc3561bf0b41ca2fd365157a0c6952bcc5a9d38b92cffc16 Jul 27 20:15:00.342144 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.342120 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8"] Jul 27 20:15:00.344425 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:15:00.344401 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9298fa_ca59_4153_a89b_f794b5da8b0e.slice/crio-ec1c692f4d3923f0948d158fab78e2fb66b2cdf33723c3e73238d7695e7690a4 WatchSource:0}: Error finding container ec1c692f4d3923f0948d158fab78e2fb66b2cdf33723c3e73238d7695e7690a4: Status 404 returned error can't find the container with id ec1c692f4d3923f0948d158fab78e2fb66b2cdf33723c3e73238d7695e7690a4 Jul 27 20:15:00.930935 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.930902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" event={"ID":"a3e74419-6790-4a83-a3ff-c4cbd0f4465d","Type":"ContainerStarted","Data":"074c6bdc993b82e917aaa0509bc10e8999150ef445d633ffe999d232528fb9b5"} Jul 27 20:15:00.931392 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.930961 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" event={"ID":"a3e74419-6790-4a83-a3ff-c4cbd0f4465d","Type":"ContainerStarted","Data":"d6265f06a8a67652fc3561bf0b41ca2fd365157a0c6952bcc5a9d38b92cffc16"} Jul 27 20:15:00.935778 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.935738 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8" event={"ID":"ac9298fa-ca59-4153-a89b-f794b5da8b0e","Type":"ContainerStarted","Data":"b2f4cabed521d6dc21d8bff064c702eab1b09e86c184ded19d7fc42a769db828"} Jul 27 20:15:00.935911 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.935793 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8" event={"ID":"ac9298fa-ca59-4153-a89b-f794b5da8b0e","Type":"ContainerStarted","Data":"ec1c692f4d3923f0948d158fab78e2fb66b2cdf33723c3e73238d7695e7690a4"} Jul 27 20:15:00.965102 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.965040 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" podStartSLOduration=0.965024244 podStartE2EDuration="965.024244ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:15:00.948403831 +0000 UTC m=+2105.560988416" watchObservedRunningTime="2026-07-27 20:15:00.965024244 +0000 UTC m=+2105.577608827" Jul 27 20:15:00.965271 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:00.965118 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8" podStartSLOduration=0.965113228 podStartE2EDuration="965.113228ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:15:00.962828549 +0000 UTC m=+2105.575413169" watchObservedRunningTime="2026-07-27 20:15:00.965113228 +0000 UTC m=+2105.577697808" Jul 27 20:15:01.760787 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:01.760754 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:15:01.816750 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:01.816712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:15:01.942201 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:01.942167 2573 generic.go:356] "Generic (PLEG): container finished" podID="ac9298fa-ca59-4153-a89b-f794b5da8b0e" containerID="b2f4cabed521d6dc21d8bff064c702eab1b09e86c184ded19d7fc42a769db828" exitCode=0 Jul 27 20:15:01.942661 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:01.942254 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8" event={"ID":"ac9298fa-ca59-4153-a89b-f794b5da8b0e","Type":"ContainerDied","Data":"b2f4cabed521d6dc21d8bff064c702eab1b09e86c184ded19d7fc42a769db828"} Jul 27 20:15:01.943708 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:01.943687 2573 generic.go:356] "Generic (PLEG): container finished" podID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerID="074c6bdc993b82e917aaa0509bc10e8999150ef445d633ffe999d232528fb9b5" exitCode=7 Jul 27 20:15:01.943833 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:01.943777 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" event={"ID":"a3e74419-6790-4a83-a3ff-c4cbd0f4465d","Type":"ContainerDied","Data":"074c6bdc993b82e917aaa0509bc10e8999150ef445d633ffe999d232528fb9b5"} Jul 27 20:15:01.943895 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:01.943883 2573 scope.go:117] "RemoveContainer" containerID="074c6bdc993b82e917aaa0509bc10e8999150ef445d633ffe999d232528fb9b5" Jul 27 20:15:02.228840 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:02.228805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:15:02.885090 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:02.884995 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:15:02.885902 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:02.885880 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:15:02.952428 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:02.952392 2573 generic.go:356] "Generic (PLEG): container finished" podID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerID="092e74ee54c7a511b238dc876ebd5ae9133d313305a1296ae795b7f20c758905" exitCode=7 Jul 27 20:15:02.952815 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:02.952482 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" event={"ID":"a3e74419-6790-4a83-a3ff-c4cbd0f4465d","Type":"ContainerDied","Data":"092e74ee54c7a511b238dc876ebd5ae9133d313305a1296ae795b7f20c758905"} Jul 27 20:15:02.952815 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:02.952541 2573 scope.go:117] "RemoveContainer" containerID="074c6bdc993b82e917aaa0509bc10e8999150ef445d633ffe999d232528fb9b5" Jul 27 20:15:02.952815 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:02.952696 2573 scope.go:117] "RemoveContainer" containerID="092e74ee54c7a511b238dc876ebd5ae9133d313305a1296ae795b7f20c758905" Jul 27 20:15:02.953235 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:15:02.952954 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf_odh-ai-gateway-infra(a3e74419-6790-4a83-a3ff-c4cbd0f4465d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" Jul 27 20:15:03.086191 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:03.086166 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8" Jul 27 20:15:03.959515 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:03.959488 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8" Jul 27 20:15:03.959972 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:03.959510 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-hjnc8" event={"ID":"ac9298fa-ca59-4153-a89b-f794b5da8b0e","Type":"ContainerDied","Data":"ec1c692f4d3923f0948d158fab78e2fb66b2cdf33723c3e73238d7695e7690a4"} Jul 27 20:15:03.959972 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:03.959560 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec1c692f4d3923f0948d158fab78e2fb66b2cdf33723c3e73238d7695e7690a4" Jul 27 20:15:03.961394 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:03.961366 2573 scope.go:117] "RemoveContainer" containerID="092e74ee54c7a511b238dc876ebd5ae9133d313305a1296ae795b7f20c758905" Jul 27 20:15:03.961701 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:15:03.961677 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf_odh-ai-gateway-infra(a3e74419-6790-4a83-a3ff-c4cbd0f4465d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" Jul 27 20:15:10.692201 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:10.692164 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:15:10.938192 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:10.938155 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:15:10.938377 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:10.938343 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:15:13.437145 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:13.437103 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:15:17.869774 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:17.869731 2573 scope.go:117] "RemoveContainer" containerID="092e74ee54c7a511b238dc876ebd5ae9133d313305a1296ae795b7f20c758905" Jul 27 20:15:19.045712 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:19.045676 2573 generic.go:356] "Generic (PLEG): container finished" podID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerID="5c5dbc61cb7c8107420f821c937a71fc284bc78ac9c8313f34567100917177d8" exitCode=0 Jul 27 20:15:19.046120 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:19.045750 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" event={"ID":"a3e74419-6790-4a83-a3ff-c4cbd0f4465d","Type":"ContainerDied","Data":"5c5dbc61cb7c8107420f821c937a71fc284bc78ac9c8313f34567100917177d8"} Jul 27 20:15:19.046120 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:19.045815 2573 scope.go:117] "RemoveContainer" containerID="092e74ee54c7a511b238dc876ebd5ae9133d313305a1296ae795b7f20c758905" Jul 27 20:15:20.083648 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:20.083610 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf"] Jul 27 20:15:20.191493 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:20.191467 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" Jul 27 20:15:21.060160 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:21.060127 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" Jul 27 20:15:21.060734 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:21.060177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf" event={"ID":"a3e74419-6790-4a83-a3ff-c4cbd0f4465d","Type":"ContainerDied","Data":"d6265f06a8a67652fc3561bf0b41ca2fd365157a0c6952bcc5a9d38b92cffc16"} Jul 27 20:15:21.060734 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:21.060270 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6265f06a8a67652fc3561bf0b41ca2fd365157a0c6952bcc5a9d38b92cffc16" Jul 27 20:15:21.092524 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:21.092489 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf"] Jul 27 20:15:21.098013 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:21.097985 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0a5392-29753055-5k9xf"] Jul 27 20:15:21.875805 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:21.875766 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" path="/var/lib/kubelet/pods/a3e74419-6790-4a83-a3ff-c4cbd0f4465d/volumes" Jul 27 20:15:29.936992 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:29.936950 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:15:30.387069 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:30.386973 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:15:38.168163 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:38.168111 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:15:42.035748 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:15:42.035714 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:16:11.414599 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:11.414567 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:12.260494 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.260461 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:12.381584 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.381550 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l"] Jul 27 20:16:12.381926 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.381914 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerName="cleanup" Jul 27 20:16:12.381970 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.381929 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerName="cleanup" Jul 27 20:16:12.381970 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.381939 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerName="cleanup" Jul 27 20:16:12.381970 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.381944 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerName="cleanup" Jul 27 20:16:12.381970 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.381965 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9298fa-ca59-4153-a89b-f794b5da8b0e" containerName="cleanup" Jul 27 20:16:12.381970 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.381970 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9298fa-ca59-4153-a89b-f794b5da8b0e" containerName="cleanup" Jul 27 20:16:12.382126 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.382030 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerName="cleanup" Jul 27 20:16:12.382126 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.382037 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerName="cleanup" Jul 27 20:16:12.382126 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.382043 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerName="cleanup" Jul 27 20:16:12.382126 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.382053 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac9298fa-ca59-4153-a89b-f794b5da8b0e" containerName="cleanup" Jul 27 20:16:12.385162 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.385146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" Jul 27 20:16:12.387935 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.387912 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:16:12.387935 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.387925 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:16:12.392502 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.392480 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l"] Jul 27 20:16:12.441345 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.441281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b8324d-04cb-44b8-a040-3c8f6aadc3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l\" (UID: \"41b8324d-04cb-44b8-a040-3c8f6aadc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" Jul 27 20:16:12.542547 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.542454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b8324d-04cb-44b8-a040-3c8f6aadc3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l\" (UID: \"41b8324d-04cb-44b8-a040-3c8f6aadc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" Jul 27 20:16:12.543084 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.543063 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b8324d-04cb-44b8-a040-3c8f6aadc3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l\" (UID: \"41b8324d-04cb-44b8-a040-3c8f6aadc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" Jul 27 20:16:12.697479 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:12.697440 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" Jul 27 20:16:13.042639 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:13.042612 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l"] Jul 27 20:16:13.044372 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:16:13.044347 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b8324d_04cb_44b8_a040_3c8f6aadc3ff.slice/crio-b0fdcc2cd24780e1e1d12b2435664470f85bba9b6b080cc4b92ec0a9c08a05b5 WatchSource:0}: Error finding container b0fdcc2cd24780e1e1d12b2435664470f85bba9b6b080cc4b92ec0a9c08a05b5: Status 404 returned error can't find the container with id b0fdcc2cd24780e1e1d12b2435664470f85bba9b6b080cc4b92ec0a9c08a05b5 Jul 27 20:16:13.353057 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:13.353022 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" event={"ID":"41b8324d-04cb-44b8-a040-3c8f6aadc3ff","Type":"ContainerStarted","Data":"50e397098d4a1ec50dfb746f24a5cf7a8f56af3813c00a3fde582d3ab8a50866"} Jul 27 20:16:13.353249 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:13.353074 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" event={"ID":"41b8324d-04cb-44b8-a040-3c8f6aadc3ff","Type":"ContainerStarted","Data":"b0fdcc2cd24780e1e1d12b2435664470f85bba9b6b080cc4b92ec0a9c08a05b5"} Jul 27 20:16:13.369590 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:13.369538 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" podStartSLOduration=1.3695219650000001 podStartE2EDuration="1.369521965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:16:13.368957479 +0000 UTC m=+2177.981542081" watchObservedRunningTime="2026-07-27 20:16:13.369521965 +0000 UTC m=+2177.982106772" Jul 27 20:16:14.359892 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:14.359859 2573 generic.go:356] "Generic (PLEG): container finished" podID="41b8324d-04cb-44b8-a040-3c8f6aadc3ff" containerID="50e397098d4a1ec50dfb746f24a5cf7a8f56af3813c00a3fde582d3ab8a50866" exitCode=0 Jul 27 20:16:14.360263 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:14.359950 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" event={"ID":"41b8324d-04cb-44b8-a040-3c8f6aadc3ff","Type":"ContainerDied","Data":"50e397098d4a1ec50dfb746f24a5cf7a8f56af3813c00a3fde582d3ab8a50866"} Jul 27 20:16:15.500252 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:15.500225 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" Jul 27 20:16:15.572205 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:15.572163 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b8324d-04cb-44b8-a040-3c8f6aadc3ff-maas-api-service-ca\") pod \"41b8324d-04cb-44b8-a040-3c8f6aadc3ff\" (UID: \"41b8324d-04cb-44b8-a040-3c8f6aadc3ff\") " Jul 27 20:16:15.572606 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:15.572584 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b8324d-04cb-44b8-a040-3c8f6aadc3ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41b8324d-04cb-44b8-a040-3c8f6aadc3ff" (UID: "41b8324d-04cb-44b8-a040-3c8f6aadc3ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:16:15.673034 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:15.672955 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b8324d-04cb-44b8-a040-3c8f6aadc3ff-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:16:16.339375 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:16.339341 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:16:16.378762 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:16.378733 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" Jul 27 20:16:16.378919 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:16.378779 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l" event={"ID":"41b8324d-04cb-44b8-a040-3c8f6aadc3ff","Type":"ContainerDied","Data":"b0fdcc2cd24780e1e1d12b2435664470f85bba9b6b080cc4b92ec0a9c08a05b5"} Jul 27 20:16:16.378919 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:16.378822 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0fdcc2cd24780e1e1d12b2435664470f85bba9b6b080cc4b92ec0a9c08a05b5" Jul 27 20:16:23.027041 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.027003 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:23.063051 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.063022 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq"] Jul 27 20:16:23.063274 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.063232 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" podUID="1e8a35a7-4637-4320-9c25-92a092822e29" containerName="payload-processing" containerID="cri-o://93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8" gracePeriod=30 Jul 27 20:16:23.070608 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.070581 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66"] Jul 27 20:16:23.071016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.070982 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" podUID="dffd584f-4bd7-4b00-8c48-c7e85c827646" containerName="payload-pre-processing" containerID="cri-o://1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca" gracePeriod=30 Jul 27 20:16:23.116417 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.116350 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:23.180762 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.180729 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:23.324846 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.324818 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:16:23.352202 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.352178 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:16:23.424180 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.424142 2573 generic.go:356] "Generic (PLEG): container finished" podID="dffd584f-4bd7-4b00-8c48-c7e85c827646" containerID="1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca" exitCode=0 Jul 27 20:16:23.424385 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.424183 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" event={"ID":"dffd584f-4bd7-4b00-8c48-c7e85c827646","Type":"ContainerDied","Data":"1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca"} Jul 27 20:16:23.424385 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.424232 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" event={"ID":"dffd584f-4bd7-4b00-8c48-c7e85c827646","Type":"ContainerDied","Data":"00e3a425e9925058a9c85236118af5ed6c065478f3d32ea7ae7d0f95e5bbcc6e"} Jul 27 20:16:23.424385 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.424255 2573 scope.go:117] "RemoveContainer" containerID="1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca" Jul 27 20:16:23.424385 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.424187 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66" Jul 27 20:16:23.425737 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.425701 2573 generic.go:356] "Generic (PLEG): container finished" podID="1e8a35a7-4637-4320-9c25-92a092822e29" containerID="93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8" exitCode=0 Jul 27 20:16:23.425844 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.425745 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" event={"ID":"1e8a35a7-4637-4320-9c25-92a092822e29","Type":"ContainerDied","Data":"93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8"} Jul 27 20:16:23.425844 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.425773 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" event={"ID":"1e8a35a7-4637-4320-9c25-92a092822e29","Type":"ContainerDied","Data":"47ac0e8ffb5a8bd2e256427f251688c84c6ab0bd4e049e31e2b62866e51b626f"} Jul 27 20:16:23.425944 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.425897 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq" Jul 27 20:16:23.433054 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.433038 2573 scope.go:117] "RemoveContainer" containerID="1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca" Jul 27 20:16:23.433352 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:16:23.433334 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca\": container with ID starting with 1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca not found: ID does not exist" containerID="1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca" Jul 27 20:16:23.433419 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.433361 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca"} err="failed to get container status \"1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca\": rpc error: code = NotFound desc = could not find container \"1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca\": container with ID starting with 1dc00da317228f97a232efe136e2a32c6ca1b139a3d3b04496d4c4483f78a9ca not found: ID does not exist" Jul 27 20:16:23.433419 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.433379 2573 scope.go:117] "RemoveContainer" containerID="93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8" Jul 27 20:16:23.441372 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.441354 2573 scope.go:117] "RemoveContainer" containerID="93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8" Jul 27 20:16:23.441606 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:16:23.441590 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8\": container with ID starting with 93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8 not found: ID does not exist" containerID="93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8" Jul 27 20:16:23.441687 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.441611 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8"} err="failed to get container status \"93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8\": rpc error: code = NotFound desc = could not find container \"93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8\": container with ID starting with 93c71e7de320dd4a08658413e9b7d4a5aae104038152cfc56cc5e53007c030c8 not found: ID does not exist" Jul 27 20:16:23.442855 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.442840 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a35a7-4637-4320-9c25-92a092822e29-plugins-config-volume\") pod \"1e8a35a7-4637-4320-9c25-92a092822e29\" (UID: \"1e8a35a7-4637-4320-9c25-92a092822e29\") " Jul 27 20:16:23.442917 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.442867 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpj7v\" (UniqueName: \"kubernetes.io/projected/dffd584f-4bd7-4b00-8c48-c7e85c827646-kube-api-access-qpj7v\") pod \"dffd584f-4bd7-4b00-8c48-c7e85c827646\" (UID: \"dffd584f-4bd7-4b00-8c48-c7e85c827646\") " Jul 27 20:16:23.442917 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.442889 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffd584f-4bd7-4b00-8c48-c7e85c827646-plugins-config-volume\") pod \"dffd584f-4bd7-4b00-8c48-c7e85c827646\" (UID: \"dffd584f-4bd7-4b00-8c48-c7e85c827646\") " Jul 27 20:16:23.442917 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.442911 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44swx\" (UniqueName: \"kubernetes.io/projected/1e8a35a7-4637-4320-9c25-92a092822e29-kube-api-access-44swx\") pod \"1e8a35a7-4637-4320-9c25-92a092822e29\" (UID: \"1e8a35a7-4637-4320-9c25-92a092822e29\") " Jul 27 20:16:23.443259 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.443227 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8a35a7-4637-4320-9c25-92a092822e29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e8a35a7-4637-4320-9c25-92a092822e29" (UID: "1e8a35a7-4637-4320-9c25-92a092822e29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:16:23.443377 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.443258 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffd584f-4bd7-4b00-8c48-c7e85c827646-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dffd584f-4bd7-4b00-8c48-c7e85c827646" (UID: "dffd584f-4bd7-4b00-8c48-c7e85c827646"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:16:23.445004 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.444986 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8a35a7-4637-4320-9c25-92a092822e29-kube-api-access-44swx" (OuterVolumeSpecName: "kube-api-access-44swx") pod "1e8a35a7-4637-4320-9c25-92a092822e29" (UID: "1e8a35a7-4637-4320-9c25-92a092822e29"). InnerVolumeSpecName "kube-api-access-44swx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:16:23.445083 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.445037 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffd584f-4bd7-4b00-8c48-c7e85c827646-kube-api-access-qpj7v" (OuterVolumeSpecName: "kube-api-access-qpj7v") pod "dffd584f-4bd7-4b00-8c48-c7e85c827646" (UID: "dffd584f-4bd7-4b00-8c48-c7e85c827646"). InnerVolumeSpecName "kube-api-access-qpj7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:16:23.544870 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.544781 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a35a7-4637-4320-9c25-92a092822e29-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:16:23.544870 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.544823 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpj7v\" (UniqueName: \"kubernetes.io/projected/dffd584f-4bd7-4b00-8c48-c7e85c827646-kube-api-access-qpj7v\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:16:23.544870 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.544838 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffd584f-4bd7-4b00-8c48-c7e85c827646-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:16:23.544870 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.544852 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44swx\" (UniqueName: \"kubernetes.io/projected/1e8a35a7-4637-4320-9c25-92a092822e29-kube-api-access-44swx\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:16:23.758203 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.758175 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66"] Jul 27 20:16:23.768221 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.768195 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-849f6b-5bc4d659d8-wbh66"] Jul 27 20:16:23.789685 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.789656 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq"] Jul 27 20:16:23.804632 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.804568 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-849f6b-55c86ff668-tvjjq"] Jul 27 20:16:23.876422 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.876380 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e8a35a7-4637-4320-9c25-92a092822e29" path="/var/lib/kubelet/pods/1e8a35a7-4637-4320-9c25-92a092822e29/volumes" Jul 27 20:16:23.876840 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:23.876820 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dffd584f-4bd7-4b00-8c48-c7e85c827646" path="/var/lib/kubelet/pods/dffd584f-4bd7-4b00-8c48-c7e85c827646/volumes" Jul 27 20:16:27.461502 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:27.461466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:27.467917 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:27.467889 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l"] Jul 27 20:16:27.472565 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:27.472543 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-849f6b-7k85l"] Jul 27 20:16:27.862160 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:27.862071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:27.875263 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:27.875230 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41b8324d-04cb-44b8-a040-3c8f6aadc3ff" path="/var/lib/kubelet/pods/41b8324d-04cb-44b8-a040-3c8f6aadc3ff/volumes" Jul 27 20:16:28.835661 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:28.835625 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:16:31.519360 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:31.519328 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:31.585650 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:31.585618 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:33.857459 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:33.857420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:33.957694 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:33.957654 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:34.126680 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:34.126582 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:34.392037 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:34.391946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:44.933619 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:44.933579 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:44.963398 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:44.958263 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c"] Jul 27 20:16:44.963398 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:44.958659 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" podUID="698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7" containerName="payload-processing" containerID="cri-o://7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d" gracePeriod=30 Jul 27 20:16:44.970213 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:44.969673 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm"] Jul 27 20:16:44.970213 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:44.969925 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" podUID="e54198f9-fc95-4003-abbf-a2413e25e331" containerName="payload-pre-processing" containerID="cri-o://d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b" gracePeriod=30 Jul 27 20:16:45.006224 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.006189 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:45.073461 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.073421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:45.251624 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.251592 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:16:45.254484 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.254467 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:16:45.333927 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.333894 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s85jk\" (UniqueName: \"kubernetes.io/projected/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-kube-api-access-s85jk\") pod \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\" (UID: \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\") " Jul 27 20:16:45.333927 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.333927 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5lfw\" (UniqueName: \"kubernetes.io/projected/e54198f9-fc95-4003-abbf-a2413e25e331-kube-api-access-c5lfw\") pod \"e54198f9-fc95-4003-abbf-a2413e25e331\" (UID: \"e54198f9-fc95-4003-abbf-a2413e25e331\") " Jul 27 20:16:45.334151 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.333979 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-plugins-config-volume\") pod \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\" (UID: \"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7\") " Jul 27 20:16:45.334151 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.334015 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54198f9-fc95-4003-abbf-a2413e25e331-plugins-config-volume\") pod \"e54198f9-fc95-4003-abbf-a2413e25e331\" (UID: \"e54198f9-fc95-4003-abbf-a2413e25e331\") " Jul 27 20:16:45.334446 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.334415 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7" (UID: "698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:16:45.334573 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.334463 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54198f9-fc95-4003-abbf-a2413e25e331-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e54198f9-fc95-4003-abbf-a2413e25e331" (UID: "e54198f9-fc95-4003-abbf-a2413e25e331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:16:45.336188 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.336168 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54198f9-fc95-4003-abbf-a2413e25e331-kube-api-access-c5lfw" (OuterVolumeSpecName: "kube-api-access-c5lfw") pod "e54198f9-fc95-4003-abbf-a2413e25e331" (UID: "e54198f9-fc95-4003-abbf-a2413e25e331"). InnerVolumeSpecName "kube-api-access-c5lfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:16:45.336266 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.336190 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-kube-api-access-s85jk" (OuterVolumeSpecName: "kube-api-access-s85jk") pod "698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7" (UID: "698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7"). InnerVolumeSpecName "kube-api-access-s85jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:16:45.435502 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.435465 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54198f9-fc95-4003-abbf-a2413e25e331-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:16:45.435502 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.435493 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85jk\" (UniqueName: \"kubernetes.io/projected/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-kube-api-access-s85jk\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:16:45.435502 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.435503 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5lfw\" (UniqueName: \"kubernetes.io/projected/e54198f9-fc95-4003-abbf-a2413e25e331-kube-api-access-c5lfw\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:16:45.435502 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.435512 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:16:45.554484 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.554389 2573 generic.go:356] "Generic (PLEG): container finished" podID="e54198f9-fc95-4003-abbf-a2413e25e331" containerID="d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b" exitCode=0 Jul 27 20:16:45.554484 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.554441 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" Jul 27 20:16:45.554728 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.554482 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" event={"ID":"e54198f9-fc95-4003-abbf-a2413e25e331","Type":"ContainerDied","Data":"d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b"} Jul 27 20:16:45.554728 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.554531 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm" event={"ID":"e54198f9-fc95-4003-abbf-a2413e25e331","Type":"ContainerDied","Data":"83be414bc8e8e8e4306dbd87f4e8c574c12316c9461f6d04d47644f4d3fcd7a1"} Jul 27 20:16:45.554728 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.554546 2573 scope.go:117] "RemoveContainer" containerID="d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b" Jul 27 20:16:45.556013 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.555988 2573 generic.go:356] "Generic (PLEG): container finished" podID="698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7" containerID="7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d" exitCode=0 Jul 27 20:16:45.556122 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.556045 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" Jul 27 20:16:45.556210 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.556064 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" event={"ID":"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7","Type":"ContainerDied","Data":"7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d"} Jul 27 20:16:45.556254 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.556240 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c" event={"ID":"698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7","Type":"ContainerDied","Data":"051ecf436e074adaf865b9c0dc8b69aa2cb46b02f6f1fb8f8e3599aea5458d6c"} Jul 27 20:16:45.563981 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.563960 2573 scope.go:117] "RemoveContainer" containerID="d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b" Jul 27 20:16:45.564242 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:16:45.564225 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b\": container with ID starting with d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b not found: ID does not exist" containerID="d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b" Jul 27 20:16:45.564339 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.564255 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b"} err="failed to get container status \"d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b\": rpc error: code = NotFound desc = could not find container \"d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b\": container with ID starting with d04ed6c4569932f12c82b9d871dfba018576181c16ce96270acfe7f8999eed6b not found: ID does not exist" Jul 27 20:16:45.564339 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.564313 2573 scope.go:117] "RemoveContainer" containerID="7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d" Jul 27 20:16:45.572636 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.572620 2573 scope.go:117] "RemoveContainer" containerID="7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d" Jul 27 20:16:45.572872 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:16:45.572854 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d\": container with ID starting with 7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d not found: ID does not exist" containerID="7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d" Jul 27 20:16:45.572936 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.572878 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d"} err="failed to get container status \"7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d\": rpc error: code = NotFound desc = could not find container \"7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d\": container with ID starting with 7cac7cdd3fcd3166a2bc53697cc1aa204665f118b1a32f732510132a150dd51d not found: ID does not exist" Jul 27 20:16:45.579602 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.579580 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c"] Jul 27 20:16:45.583843 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.583816 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a5392-5b9b65c8dd-sbw2c"] Jul 27 20:16:45.594044 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.594021 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm"] Jul 27 20:16:45.597936 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.597915 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a5392-7f459d6476-57xfm"] Jul 27 20:16:45.876061 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.875981 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7" path="/var/lib/kubelet/pods/698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7/volumes" Jul 27 20:16:45.876526 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:45.876504 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e54198f9-fc95-4003-abbf-a2413e25e331" path="/var/lib/kubelet/pods/e54198f9-fc95-4003-abbf-a2413e25e331/volumes" Jul 27 20:16:47.754124 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:47.754089 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:16:49.601019 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:49.600980 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:49.982422 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:49.982388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:55.935777 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:55.935728 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:56.009539 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:56.009500 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:56.130547 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:56.130517 2573 scope.go:117] "RemoveContainer" containerID="55d4ccfd8b364f72af6f23a6e28fe82c3b9488fa264099e7989d9f057e4e544a" Jul 27 20:16:56.208361 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:56.208250 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:56.434514 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:56.434482 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:56.775844 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:56.775799 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:57.098905 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:57.098827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:16:57.368985 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:16:57.368904 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:00.474031 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:00.473994 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:00.664730 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:00.664701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:00.763579 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:00.763492 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:00.862590 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:00.862550 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:01.287902 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.287868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:01.433165 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.433119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:01.491461 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491425 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr"] Jul 27 20:17:01.491848 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491833 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7" containerName="payload-processing" Jul 27 20:17:01.491893 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491850 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7" containerName="payload-processing" Jul 27 20:17:01.491893 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491864 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54198f9-fc95-4003-abbf-a2413e25e331" containerName="payload-pre-processing" Jul 27 20:17:01.491893 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491869 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54198f9-fc95-4003-abbf-a2413e25e331" containerName="payload-pre-processing" Jul 27 20:17:01.491893 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491880 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8a35a7-4637-4320-9c25-92a092822e29" containerName="payload-processing" Jul 27 20:17:01.491893 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491886 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8a35a7-4637-4320-9c25-92a092822e29" containerName="payload-processing" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491896 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41b8324d-04cb-44b8-a040-3c8f6aadc3ff" containerName="revoke-keys" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491902 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41b8324d-04cb-44b8-a040-3c8f6aadc3ff" containerName="revoke-keys" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491908 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerName="cleanup" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491912 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e74419-6790-4a83-a3ff-c4cbd0f4465d" containerName="cleanup" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491920 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffd584f-4bd7-4b00-8c48-c7e85c827646" containerName="payload-pre-processing" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491925 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffd584f-4bd7-4b00-8c48-c7e85c827646" containerName="payload-pre-processing" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491988 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e54198f9-fc95-4003-abbf-a2413e25e331" containerName="payload-pre-processing" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.491996 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dffd584f-4bd7-4b00-8c48-c7e85c827646" containerName="payload-pre-processing" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.492003 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="698cd6d7-52a5-4ee9-8a59-c9b0dc46c5d7" containerName="payload-processing" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.492009 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41b8324d-04cb-44b8-a040-3c8f6aadc3ff" containerName="revoke-keys" Jul 27 20:17:01.492052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.492017 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8a35a7-4637-4320-9c25-92a092822e29" containerName="payload-processing" Jul 27 20:17:01.495106 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.495086 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:01.498008 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.497984 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d453a8-dockercfg-hnfpm\"" Jul 27 20:17:01.498390 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.498376 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d453a8\"" Jul 27 20:17:01.507207 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.507184 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr"] Jul 27 20:17:01.576875 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.576781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glp6d\" (UniqueName: \"kubernetes.io/projected/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-kube-api-access-glp6d\") pod \"payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr\" (UID: \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:01.577126 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.576950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr\" (UID: \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:01.579519 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.579498 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh"] Jul 27 20:17:01.583026 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.583007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:01.593415 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.593391 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh"] Jul 27 20:17:01.677752 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.677724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr\" (UID: \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:01.677944 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.677782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glp6d\" (UniqueName: \"kubernetes.io/projected/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-kube-api-access-glp6d\") pod \"payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr\" (UID: \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:01.677944 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.677803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh\" (UID: \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\") " pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:01.677944 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.677857 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxj8\" (UniqueName: \"kubernetes.io/projected/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-kube-api-access-skxj8\") pod \"payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh\" (UID: \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\") " pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:01.678411 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.678393 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr\" (UID: \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:01.686233 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.686215 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glp6d\" (UniqueName: \"kubernetes.io/projected/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-kube-api-access-glp6d\") pod \"payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr\" (UID: \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:01.779215 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.779187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxj8\" (UniqueName: \"kubernetes.io/projected/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-kube-api-access-skxj8\") pod \"payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh\" (UID: \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\") " pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:01.779437 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.779275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh\" (UID: \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\") " pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:01.779950 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.779924 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh\" (UID: \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\") " pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:01.789211 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.789192 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxj8\" (UniqueName: \"kubernetes.io/projected/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-kube-api-access-skxj8\") pod \"payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh\" (UID: \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\") " pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:01.806055 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.806030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:01.894549 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.894466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:01.945751 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:01.945727 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr"] Jul 27 20:17:01.948579 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:17:01.948533 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe138112_de66_45e5_a4fd_27bb1fd5a6c2.slice/crio-e96c7374c139bc9ea6944c57515700ed23d3bc3037b9b3bffb3c684557039a25 WatchSource:0}: Error finding container e96c7374c139bc9ea6944c57515700ed23d3bc3037b9b3bffb3c684557039a25: Status 404 returned error can't find the container with id e96c7374c139bc9ea6944c57515700ed23d3bc3037b9b3bffb3c684557039a25 Jul 27 20:17:02.039230 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:02.039203 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh"] Jul 27 20:17:02.040542 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:17:02.040514 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0672e4d_5aca_4240_ae6e_10e7eeb9b962.slice/crio-d5ea805979b0fa7e3c0ee24f2fbfcaa359649df5fedbd127abe68b62bf397cb2 WatchSource:0}: Error finding container d5ea805979b0fa7e3c0ee24f2fbfcaa359649df5fedbd127abe68b62bf397cb2: Status 404 returned error can't find the container with id d5ea805979b0fa7e3c0ee24f2fbfcaa359649df5fedbd127abe68b62bf397cb2 Jul 27 20:17:02.647859 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:02.647818 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" event={"ID":"fe138112-de66-45e5-a4fd-27bb1fd5a6c2","Type":"ContainerStarted","Data":"58b0923dc02fe004bae3aa62ee3d7beef60a67e9eb65295619d3aa925fd99a70"} Jul 27 20:17:02.648274 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:02.647914 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" event={"ID":"fe138112-de66-45e5-a4fd-27bb1fd5a6c2","Type":"ContainerStarted","Data":"e96c7374c139bc9ea6944c57515700ed23d3bc3037b9b3bffb3c684557039a25"} Jul 27 20:17:02.648274 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:02.647946 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:02.649333 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:02.649289 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" event={"ID":"b0672e4d-5aca-4240-ae6e-10e7eeb9b962","Type":"ContainerStarted","Data":"9d9558ea1c916c13147006d05aae1e0290ff434719c6ecf588485c853f6a1383"} Jul 27 20:17:02.649474 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:02.649344 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" event={"ID":"b0672e4d-5aca-4240-ae6e-10e7eeb9b962","Type":"ContainerStarted","Data":"d5ea805979b0fa7e3c0ee24f2fbfcaa359649df5fedbd127abe68b62bf397cb2"} Jul 27 20:17:02.649474 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:02.649380 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:02.666097 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:02.666036 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" podStartSLOduration=1.666020879 podStartE2EDuration="1.666020879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:02.664533141 +0000 UTC m=+2227.277117743" watchObservedRunningTime="2026-07-27 20:17:02.666020879 +0000 UTC m=+2227.278605461" Jul 27 20:17:02.687056 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:02.687003 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" podStartSLOduration=1.686987456 podStartE2EDuration="1.686987456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:02.685607216 +0000 UTC m=+2227.298191845" watchObservedRunningTime="2026-07-27 20:17:02.686987456 +0000 UTC m=+2227.299572038" Jul 27 20:17:04.944010 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:04.943978 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:05.443607 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:05.443573 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:05.710569 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:05.710490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:05.934351 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:05.934318 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:07.736312 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:07.736275 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:17:08.585656 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:08.585621 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:08.683080 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:08.683038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:08.784288 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:08.784255 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:08.991705 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:08.991671 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:09.448077 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.448042 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:09.549112 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.549068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:09.556614 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.556575 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78"] Jul 27 20:17:09.560338 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.560315 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:09.563348 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.563326 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-83e699-dockercfg-825jh\"" Jul 27 20:17:09.563464 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.563326 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-83e699\"" Jul 27 20:17:09.572513 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.572489 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78"] Jul 27 20:17:09.633953 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.633922 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv"] Jul 27 20:17:09.637426 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.637408 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:09.649133 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.649104 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv"] Jul 27 20:17:09.649485 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.649463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319db771-873c-4039-8196-1206d4bfc1bd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-83e699-67f886f846-t5b78\" (UID: \"319db771-873c-4039-8196-1206d4bfc1bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:09.649610 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.649505 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxbx\" (UniqueName: \"kubernetes.io/projected/319db771-873c-4039-8196-1206d4bfc1bd-kube-api-access-vwxbx\") pod \"payload-processing-e2e-api-b-83e699-67f886f846-t5b78\" (UID: \"319db771-873c-4039-8196-1206d4bfc1bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:09.750537 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.750422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kkrp\" (UniqueName: \"kubernetes.io/projected/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-kube-api-access-4kkrp\") pod \"payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv\" (UID: \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:09.750537 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.750488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv\" (UID: \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:09.750781 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.750571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319db771-873c-4039-8196-1206d4bfc1bd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-83e699-67f886f846-t5b78\" (UID: \"319db771-873c-4039-8196-1206d4bfc1bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:09.750781 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.750619 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxbx\" (UniqueName: \"kubernetes.io/projected/319db771-873c-4039-8196-1206d4bfc1bd-kube-api-access-vwxbx\") pod \"payload-processing-e2e-api-b-83e699-67f886f846-t5b78\" (UID: \"319db771-873c-4039-8196-1206d4bfc1bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:09.751221 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.751199 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319db771-873c-4039-8196-1206d4bfc1bd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-83e699-67f886f846-t5b78\" (UID: \"319db771-873c-4039-8196-1206d4bfc1bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:09.759396 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.759372 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxbx\" (UniqueName: \"kubernetes.io/projected/319db771-873c-4039-8196-1206d4bfc1bd-kube-api-access-vwxbx\") pod \"payload-processing-e2e-api-b-83e699-67f886f846-t5b78\" (UID: \"319db771-873c-4039-8196-1206d4bfc1bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:09.851671 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.851635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kkrp\" (UniqueName: \"kubernetes.io/projected/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-kube-api-access-4kkrp\") pod \"payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv\" (UID: \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:09.852134 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.851685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv\" (UID: \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:09.852531 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.852509 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv\" (UID: \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:09.860329 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.860287 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kkrp\" (UniqueName: \"kubernetes.io/projected/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-kube-api-access-4kkrp\") pod \"payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv\" (UID: \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:09.871119 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.871090 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:09.948323 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:09.948251 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:10.040372 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:17:10.040340 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319db771_873c_4039_8196_1206d4bfc1bd.slice/crio-93a6c6f6281535587b6819485ec3e332176a7ee167850d28d1d71f0f3d0c51db WatchSource:0}: Error finding container 93a6c6f6281535587b6819485ec3e332176a7ee167850d28d1d71f0f3d0c51db: Status 404 returned error can't find the container with id 93a6c6f6281535587b6819485ec3e332176a7ee167850d28d1d71f0f3d0c51db Jul 27 20:17:10.041276 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.041238 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78"] Jul 27 20:17:10.134448 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.134421 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv"] Jul 27 20:17:10.136020 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:17:10.135977 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6fe376_de96_4fcd_8e85_ad15fb9effe1.slice/crio-28e7a075acaa9b388bad48a82c3f2ae47b88fe2cb21b890208ba60399ec2319e WatchSource:0}: Error finding container 28e7a075acaa9b388bad48a82c3f2ae47b88fe2cb21b890208ba60399ec2319e: Status 404 returned error can't find the container with id 28e7a075acaa9b388bad48a82c3f2ae47b88fe2cb21b890208ba60399ec2319e Jul 27 20:17:10.694184 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.694146 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" event={"ID":"9e6fe376-de96-4fcd-8e85-ad15fb9effe1","Type":"ContainerStarted","Data":"6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8"} Jul 27 20:17:10.694426 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.694216 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:10.694426 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.694239 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" event={"ID":"9e6fe376-de96-4fcd-8e85-ad15fb9effe1","Type":"ContainerStarted","Data":"28e7a075acaa9b388bad48a82c3f2ae47b88fe2cb21b890208ba60399ec2319e"} Jul 27 20:17:10.695806 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.695783 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" event={"ID":"319db771-873c-4039-8196-1206d4bfc1bd","Type":"ContainerStarted","Data":"e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a"} Jul 27 20:17:10.695925 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.695829 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" event={"ID":"319db771-873c-4039-8196-1206d4bfc1bd","Type":"ContainerStarted","Data":"93a6c6f6281535587b6819485ec3e332176a7ee167850d28d1d71f0f3d0c51db"} Jul 27 20:17:10.695925 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.695872 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:10.716167 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.716117 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" podStartSLOduration=1.7161026700000002 podStartE2EDuration="1.71610267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:10.71414849 +0000 UTC m=+2235.326733106" watchObservedRunningTime="2026-07-27 20:17:10.71610267 +0000 UTC m=+2235.328687307" Jul 27 20:17:10.734702 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:10.734651 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" podStartSLOduration=1.734634796 podStartE2EDuration="1.734634796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:10.731862621 +0000 UTC m=+2235.344447237" watchObservedRunningTime="2026-07-27 20:17:10.734634796 +0000 UTC m=+2235.347219442" Jul 27 20:17:13.006785 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:13.006746 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:13.656476 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:13.656435 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:13.656665 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:13.656639 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:21.703427 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:21.703332 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" podUID="9e6fe376-de96-4fcd-8e85-ad15fb9effe1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.68:9004: connect: connection refused" Jul 27 20:17:21.704371 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:21.704347 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:17:23.244408 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:23.244375 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:17:31.702744 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:31.702713 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:17:36.663888 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.663848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:36.696062 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.696027 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh"] Jul 27 20:17:36.696255 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.696232 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" podUID="b0672e4d-5aca-4240-ae6e-10e7eeb9b962" containerName="payload-processing" containerID="cri-o://9d9558ea1c916c13147006d05aae1e0290ff434719c6ecf588485c853f6a1383" gracePeriod=30 Jul 27 20:17:36.707833 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.707805 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr"] Jul 27 20:17:36.708052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.708028 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" podUID="fe138112-de66-45e5-a4fd-27bb1fd5a6c2" containerName="payload-pre-processing" containerID="cri-o://58b0923dc02fe004bae3aa62ee3d7beef60a67e9eb65295619d3aa925fd99a70" gracePeriod=30 Jul 27 20:17:36.743220 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.743179 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:36.819963 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.819934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:36.840635 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.840601 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe138112-de66-45e5-a4fd-27bb1fd5a6c2" containerID="58b0923dc02fe004bae3aa62ee3d7beef60a67e9eb65295619d3aa925fd99a70" exitCode=0 Jul 27 20:17:36.840797 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.840676 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" event={"ID":"fe138112-de66-45e5-a4fd-27bb1fd5a6c2","Type":"ContainerDied","Data":"58b0923dc02fe004bae3aa62ee3d7beef60a67e9eb65295619d3aa925fd99a70"} Jul 27 20:17:36.842511 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.842487 2573 generic.go:356] "Generic (PLEG): container finished" podID="b0672e4d-5aca-4240-ae6e-10e7eeb9b962" containerID="9d9558ea1c916c13147006d05aae1e0290ff434719c6ecf588485c853f6a1383" exitCode=0 Jul 27 20:17:36.842619 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.842561 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" event={"ID":"b0672e4d-5aca-4240-ae6e-10e7eeb9b962","Type":"ContainerDied","Data":"9d9558ea1c916c13147006d05aae1e0290ff434719c6ecf588485c853f6a1383"} Jul 27 20:17:36.982755 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.982717 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:36.985548 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:36.985530 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:37.081031 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.081003 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glp6d\" (UniqueName: \"kubernetes.io/projected/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-kube-api-access-glp6d\") pod \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\" (UID: \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\") " Jul 27 20:17:37.081215 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.081044 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-plugins-config-volume\") pod \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\" (UID: \"fe138112-de66-45e5-a4fd-27bb1fd5a6c2\") " Jul 27 20:17:37.081215 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.081092 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-plugins-config-volume\") pod \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\" (UID: \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\") " Jul 27 20:17:37.081462 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.081439 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0672e4d-5aca-4240-ae6e-10e7eeb9b962" (UID: "b0672e4d-5aca-4240-ae6e-10e7eeb9b962"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:17:37.081538 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.081479 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe138112-de66-45e5-a4fd-27bb1fd5a6c2" (UID: "fe138112-de66-45e5-a4fd-27bb1fd5a6c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:17:37.083423 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.083398 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-kube-api-access-glp6d" (OuterVolumeSpecName: "kube-api-access-glp6d") pod "fe138112-de66-45e5-a4fd-27bb1fd5a6c2" (UID: "fe138112-de66-45e5-a4fd-27bb1fd5a6c2"). InnerVolumeSpecName "kube-api-access-glp6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:17:37.181700 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.181670 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skxj8\" (UniqueName: \"kubernetes.io/projected/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-kube-api-access-skxj8\") pod \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\" (UID: \"b0672e4d-5aca-4240-ae6e-10e7eeb9b962\") " Jul 27 20:17:37.181863 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.181854 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:17:37.181956 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.181869 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glp6d\" (UniqueName: \"kubernetes.io/projected/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-kube-api-access-glp6d\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:17:37.181956 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.181879 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe138112-de66-45e5-a4fd-27bb1fd5a6c2-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:17:37.183870 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.183846 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-kube-api-access-skxj8" (OuterVolumeSpecName: "kube-api-access-skxj8") pod "b0672e4d-5aca-4240-ae6e-10e7eeb9b962" (UID: "b0672e4d-5aca-4240-ae6e-10e7eeb9b962"). InnerVolumeSpecName "kube-api-access-skxj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:17:37.282445 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.282369 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxj8\" (UniqueName: \"kubernetes.io/projected/b0672e4d-5aca-4240-ae6e-10e7eeb9b962-kube-api-access-skxj8\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:17:37.850371 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.850331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" event={"ID":"fe138112-de66-45e5-a4fd-27bb1fd5a6c2","Type":"ContainerDied","Data":"e96c7374c139bc9ea6944c57515700ed23d3bc3037b9b3bffb3c684557039a25"} Jul 27 20:17:37.850371 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.850352 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr" Jul 27 20:17:37.850896 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.850392 2573 scope.go:117] "RemoveContainer" containerID="58b0923dc02fe004bae3aa62ee3d7beef60a67e9eb65295619d3aa925fd99a70" Jul 27 20:17:37.851807 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.851787 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" Jul 27 20:17:37.851884 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.851808 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh" event={"ID":"b0672e4d-5aca-4240-ae6e-10e7eeb9b962","Type":"ContainerDied","Data":"d5ea805979b0fa7e3c0ee24f2fbfcaa359649df5fedbd127abe68b62bf397cb2"} Jul 27 20:17:37.861479 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.861454 2573 scope.go:117] "RemoveContainer" containerID="9d9558ea1c916c13147006d05aae1e0290ff434719c6ecf588485c853f6a1383" Jul 27 20:17:37.879652 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.879628 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh"] Jul 27 20:17:37.881959 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.881940 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d453a8-6455fbf4bb-6mrfh"] Jul 27 20:17:37.893105 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.893072 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr"] Jul 27 20:17:37.897488 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:37.897465 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d453a8-86dcb66ffb-kqznr"] Jul 27 20:17:39.875362 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:39.875325 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0672e4d-5aca-4240-ae6e-10e7eeb9b962" path="/var/lib/kubelet/pods/b0672e4d-5aca-4240-ae6e-10e7eeb9b962/volumes" Jul 27 20:17:39.875742 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:39.875693 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe138112-de66-45e5-a4fd-27bb1fd5a6c2" path="/var/lib/kubelet/pods/fe138112-de66-45e5-a4fd-27bb1fd5a6c2/volumes" Jul 27 20:17:41.041663 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:41.041596 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:17:41.055527 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:41.055466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:41.420341 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:41.420192 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:47.380202 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.380160 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:47.481835 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.481789 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:47.644743 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.644652 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:47.855004 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.854969 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:17:47.980618 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.980588 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7"] Jul 27 20:17:47.980965 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.980953 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe138112-de66-45e5-a4fd-27bb1fd5a6c2" containerName="payload-pre-processing" Jul 27 20:17:47.981009 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.980967 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe138112-de66-45e5-a4fd-27bb1fd5a6c2" containerName="payload-pre-processing" Jul 27 20:17:47.981009 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.980975 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0672e4d-5aca-4240-ae6e-10e7eeb9b962" containerName="payload-processing" Jul 27 20:17:47.981009 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.980980 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0672e4d-5aca-4240-ae6e-10e7eeb9b962" containerName="payload-processing" Jul 27 20:17:47.981105 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.981042 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe138112-de66-45e5-a4fd-27bb1fd5a6c2" containerName="payload-pre-processing" Jul 27 20:17:47.981105 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.981051 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0672e4d-5aca-4240-ae6e-10e7eeb9b962" containerName="payload-processing" Jul 27 20:17:47.985675 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.985656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" Jul 27 20:17:47.988382 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.988358 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:17:47.988382 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.988359 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:17:47.993101 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:47.993077 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7"] Jul 27 20:17:48.077531 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:48.077498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6ddf75-1f24-4be3-873b-1d40d74d9a1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83e699-68ww7\" (UID: \"db6ddf75-1f24-4be3-873b-1d40d74d9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" Jul 27 20:17:48.177933 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:48.177892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6ddf75-1f24-4be3-873b-1d40d74d9a1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83e699-68ww7\" (UID: \"db6ddf75-1f24-4be3-873b-1d40d74d9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" Jul 27 20:17:48.178654 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:48.178628 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6ddf75-1f24-4be3-873b-1d40d74d9a1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83e699-68ww7\" (UID: \"db6ddf75-1f24-4be3-873b-1d40d74d9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" Jul 27 20:17:48.297646 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:48.297548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" Jul 27 20:17:48.430425 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:48.430227 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7"] Jul 27 20:17:48.433100 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:17:48.433073 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6ddf75_1f24_4be3_873b_1d40d74d9a1c.slice/crio-78521cf8a8fe3fd31b98474c99636c20041fd582126bfc20c43e7024627991fb WatchSource:0}: Error finding container 78521cf8a8fe3fd31b98474c99636c20041fd582126bfc20c43e7024627991fb: Status 404 returned error can't find the container with id 78521cf8a8fe3fd31b98474c99636c20041fd582126bfc20c43e7024627991fb Jul 27 20:17:48.913734 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:48.913655 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" event={"ID":"db6ddf75-1f24-4be3-873b-1d40d74d9a1c","Type":"ContainerStarted","Data":"74dbff40ec4b55e7d599d7708d80ff3aca03572014c16c1c82a1ed429c4d8133"} Jul 27 20:17:48.913734 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:48.913703 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" event={"ID":"db6ddf75-1f24-4be3-873b-1d40d74d9a1c","Type":"ContainerStarted","Data":"78521cf8a8fe3fd31b98474c99636c20041fd582126bfc20c43e7024627991fb"} Jul 27 20:17:48.935355 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:48.935290 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" podStartSLOduration=1.935275124 podStartE2EDuration="1.935275124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:48.932979234 +0000 UTC m=+2273.545563817" watchObservedRunningTime="2026-07-27 20:17:48.935275124 +0000 UTC m=+2273.547859707" Jul 27 20:17:49.926860 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:49.926759 2573 generic.go:356] "Generic (PLEG): container finished" podID="db6ddf75-1f24-4be3-873b-1d40d74d9a1c" containerID="74dbff40ec4b55e7d599d7708d80ff3aca03572014c16c1c82a1ed429c4d8133" exitCode=0 Jul 27 20:17:49.926860 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:49.926842 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" event={"ID":"db6ddf75-1f24-4be3-873b-1d40d74d9a1c","Type":"ContainerDied","Data":"74dbff40ec4b55e7d599d7708d80ff3aca03572014c16c1c82a1ed429c4d8133"} Jul 27 20:17:51.062882 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:51.062857 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" Jul 27 20:17:51.105217 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:51.105189 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6ddf75-1f24-4be3-873b-1d40d74d9a1c-maas-api-service-ca\") pod \"db6ddf75-1f24-4be3-873b-1d40d74d9a1c\" (UID: \"db6ddf75-1f24-4be3-873b-1d40d74d9a1c\") " Jul 27 20:17:51.105604 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:51.105584 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6ddf75-1f24-4be3-873b-1d40d74d9a1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db6ddf75-1f24-4be3-873b-1d40d74d9a1c" (UID: "db6ddf75-1f24-4be3-873b-1d40d74d9a1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:17:51.206703 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:51.206626 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6ddf75-1f24-4be3-873b-1d40d74d9a1c-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:17:51.939748 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:51.939717 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" Jul 27 20:17:51.939933 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:51.939767 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7" event={"ID":"db6ddf75-1f24-4be3-873b-1d40d74d9a1c","Type":"ContainerDied","Data":"78521cf8a8fe3fd31b98474c99636c20041fd582126bfc20c43e7024627991fb"} Jul 27 20:17:51.939933 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:17:51.939823 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78521cf8a8fe3fd31b98474c99636c20041fd582126bfc20c43e7024627991fb" Jul 27 20:18:03.484163 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.484132 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78"] Jul 27 20:18:03.484762 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.484399 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" podUID="319db771-873c-4039-8196-1206d4bfc1bd" containerName="payload-processing" containerID="cri-o://e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a" gracePeriod=30 Jul 27 20:18:03.493269 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.493240 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv"] Jul 27 20:18:03.493591 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.493536 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" podUID="9e6fe376-de96-4fcd-8e85-ad15fb9effe1" containerName="payload-pre-processing" containerID="cri-o://6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8" gracePeriod=30 Jul 27 20:18:03.498398 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.498279 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:03.768139 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.768119 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:18:03.770789 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.770771 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:18:03.922133 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.922098 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kkrp\" (UniqueName: \"kubernetes.io/projected/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-kube-api-access-4kkrp\") pod \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\" (UID: \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\") " Jul 27 20:18:03.922338 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.922141 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319db771-873c-4039-8196-1206d4bfc1bd-plugins-config-volume\") pod \"319db771-873c-4039-8196-1206d4bfc1bd\" (UID: \"319db771-873c-4039-8196-1206d4bfc1bd\") " Jul 27 20:18:03.922338 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.922183 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwxbx\" (UniqueName: \"kubernetes.io/projected/319db771-873c-4039-8196-1206d4bfc1bd-kube-api-access-vwxbx\") pod \"319db771-873c-4039-8196-1206d4bfc1bd\" (UID: \"319db771-873c-4039-8196-1206d4bfc1bd\") " Jul 27 20:18:03.922338 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.922223 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-plugins-config-volume\") pod \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\" (UID: \"9e6fe376-de96-4fcd-8e85-ad15fb9effe1\") " Jul 27 20:18:03.922670 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.922639 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319db771-873c-4039-8196-1206d4bfc1bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "319db771-873c-4039-8196-1206d4bfc1bd" (UID: "319db771-873c-4039-8196-1206d4bfc1bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:18:03.922670 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.922651 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e6fe376-de96-4fcd-8e85-ad15fb9effe1" (UID: "9e6fe376-de96-4fcd-8e85-ad15fb9effe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:18:03.924543 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.924519 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319db771-873c-4039-8196-1206d4bfc1bd-kube-api-access-vwxbx" (OuterVolumeSpecName: "kube-api-access-vwxbx") pod "319db771-873c-4039-8196-1206d4bfc1bd" (UID: "319db771-873c-4039-8196-1206d4bfc1bd"). InnerVolumeSpecName "kube-api-access-vwxbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:18:03.924628 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:03.924560 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-kube-api-access-4kkrp" (OuterVolumeSpecName: "kube-api-access-4kkrp") pod "9e6fe376-de96-4fcd-8e85-ad15fb9effe1" (UID: "9e6fe376-de96-4fcd-8e85-ad15fb9effe1"). InnerVolumeSpecName "kube-api-access-4kkrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:18:04.008641 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.008547 2573 generic.go:356] "Generic (PLEG): container finished" podID="319db771-873c-4039-8196-1206d4bfc1bd" containerID="e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a" exitCode=0 Jul 27 20:18:04.008641 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.008600 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" Jul 27 20:18:04.008873 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.008639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" event={"ID":"319db771-873c-4039-8196-1206d4bfc1bd","Type":"ContainerDied","Data":"e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a"} Jul 27 20:18:04.008873 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.008686 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78" event={"ID":"319db771-873c-4039-8196-1206d4bfc1bd","Type":"ContainerDied","Data":"93a6c6f6281535587b6819485ec3e332176a7ee167850d28d1d71f0f3d0c51db"} Jul 27 20:18:04.008873 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.008710 2573 scope.go:117] "RemoveContainer" containerID="e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a" Jul 27 20:18:04.010134 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.010108 2573 generic.go:356] "Generic (PLEG): container finished" podID="9e6fe376-de96-4fcd-8e85-ad15fb9effe1" containerID="6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8" exitCode=0 Jul 27 20:18:04.010268 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.010140 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" Jul 27 20:18:04.010268 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.010176 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" event={"ID":"9e6fe376-de96-4fcd-8e85-ad15fb9effe1","Type":"ContainerDied","Data":"6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8"} Jul 27 20:18:04.010268 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.010216 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv" event={"ID":"9e6fe376-de96-4fcd-8e85-ad15fb9effe1","Type":"ContainerDied","Data":"28e7a075acaa9b388bad48a82c3f2ae47b88fe2cb21b890208ba60399ec2319e"} Jul 27 20:18:04.018662 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.018647 2573 scope.go:117] "RemoveContainer" containerID="e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a" Jul 27 20:18:04.018953 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:18:04.018933 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a\": container with ID starting with e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a not found: ID does not exist" containerID="e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a" Jul 27 20:18:04.019025 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.018964 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a"} err="failed to get container status \"e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a\": rpc error: code = NotFound desc = could not find container \"e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a\": container with ID starting with e50c17b3b143293a507e171fd71f975ac51a9653cc8f4573309ca04138b61c1a not found: ID does not exist" Jul 27 20:18:04.019025 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.018987 2573 scope.go:117] "RemoveContainer" containerID="6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8" Jul 27 20:18:04.023798 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.023781 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319db771-873c-4039-8196-1206d4bfc1bd-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:18:04.023875 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.023801 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwxbx\" (UniqueName: \"kubernetes.io/projected/319db771-873c-4039-8196-1206d4bfc1bd-kube-api-access-vwxbx\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:18:04.023875 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.023811 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:18:04.023875 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.023820 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kkrp\" (UniqueName: \"kubernetes.io/projected/9e6fe376-de96-4fcd-8e85-ad15fb9effe1-kube-api-access-4kkrp\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:18:04.027275 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.027256 2573 scope.go:117] "RemoveContainer" containerID="6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8" Jul 27 20:18:04.027543 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:18:04.027526 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8\": container with ID starting with 6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8 not found: ID does not exist" containerID="6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8" Jul 27 20:18:04.027595 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.027549 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8"} err="failed to get container status \"6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8\": rpc error: code = NotFound desc = could not find container \"6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8\": container with ID starting with 6296b3c95d105a795433489d18b3021bc20180d3014f9c8cd8e084aba2fc34c8 not found: ID does not exist" Jul 27 20:18:04.035341 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.035319 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv"] Jul 27 20:18:04.038424 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.038401 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83e699-68d499685b-mvctv"] Jul 27 20:18:04.050206 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.050178 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78"] Jul 27 20:18:04.053909 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:04.053879 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83e699-67f886f846-t5b78"] Jul 27 20:18:05.307216 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:05.307180 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:05.882165 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:05.882125 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="319db771-873c-4039-8196-1206d4bfc1bd" path="/var/lib/kubelet/pods/319db771-873c-4039-8196-1206d4bfc1bd/volumes" Jul 27 20:18:05.882638 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:05.882622 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e6fe376-de96-4fcd-8e85-ad15fb9effe1" path="/var/lib/kubelet/pods/9e6fe376-de96-4fcd-8e85-ad15fb9effe1/volumes" Jul 27 20:18:08.063365 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:08.063326 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:08.107874 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:08.107839 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7"] Jul 27 20:18:08.110661 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:08.110631 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83e699-68ww7"] Jul 27 20:18:08.515483 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:08.515446 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:08.543025 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:08.542989 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:18:09.875913 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:09.875876 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db6ddf75-1f24-4be3-873b-1d40d74d9a1c" path="/var/lib/kubelet/pods/db6ddf75-1f24-4be3-873b-1d40d74d9a1c/volumes" Jul 27 20:18:14.747922 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:14.747880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:14.810576 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:14.810541 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:15.018435 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:15.018346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:15.235789 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:15.235749 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:15.591663 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:15.591615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:15.920321 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:15.920225 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:16.251593 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:16.251557 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:19.047014 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.046975 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:19.131607 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.131574 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:19.200107 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.200063 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:19.717175 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.717139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:19.819163 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.819116 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:19.853917 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.853887 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk"] Jul 27 20:18:19.854367 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.854349 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6ddf75-1f24-4be3-873b-1d40d74d9a1c" containerName="revoke-keys" Jul 27 20:18:19.854487 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.854368 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6ddf75-1f24-4be3-873b-1d40d74d9a1c" containerName="revoke-keys" Jul 27 20:18:19.854487 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.854396 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319db771-873c-4039-8196-1206d4bfc1bd" containerName="payload-processing" Jul 27 20:18:19.854487 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.854404 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="319db771-873c-4039-8196-1206d4bfc1bd" containerName="payload-processing" Jul 27 20:18:19.854487 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.854415 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e6fe376-de96-4fcd-8e85-ad15fb9effe1" containerName="payload-pre-processing" Jul 27 20:18:19.854487 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.854423 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e6fe376-de96-4fcd-8e85-ad15fb9effe1" containerName="payload-pre-processing" Jul 27 20:18:19.854796 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.854524 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="319db771-873c-4039-8196-1206d4bfc1bd" containerName="payload-processing" Jul 27 20:18:19.854796 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.854536 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6ddf75-1f24-4be3-873b-1d40d74d9a1c" containerName="revoke-keys" Jul 27 20:18:19.854796 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.854549 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e6fe376-de96-4fcd-8e85-ad15fb9effe1" containerName="payload-pre-processing" Jul 27 20:18:19.859355 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.859333 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:19.862000 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.861978 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-c3a53e\"" Jul 27 20:18:19.862113 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.862055 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-c3a53e-dockercfg-7qcfd\"" Jul 27 20:18:19.876344 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.876316 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk"] Jul 27 20:18:19.909744 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.909710 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj"] Jul 27 20:18:19.912992 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.912960 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:19.913153 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.913134 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:19.924275 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.924254 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj"] Jul 27 20:18:19.955741 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.955707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4vp\" (UniqueName: \"kubernetes.io/projected/0670c659-f08b-4285-98a4-eccdbf780045-kube-api-access-cr4vp\") pod \"payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk\" (UID: \"0670c659-f08b-4285-98a4-eccdbf780045\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:19.955937 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:19.955775 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0670c659-f08b-4285-98a4-eccdbf780045-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk\" (UID: \"0670c659-f08b-4285-98a4-eccdbf780045\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:20.057107 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.057021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj\" (UID: \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:20.057107 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.057096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2kb8\" (UniqueName: \"kubernetes.io/projected/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-kube-api-access-n2kb8\") pod \"payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj\" (UID: \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:20.057572 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.057196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4vp\" (UniqueName: \"kubernetes.io/projected/0670c659-f08b-4285-98a4-eccdbf780045-kube-api-access-cr4vp\") pod \"payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk\" (UID: \"0670c659-f08b-4285-98a4-eccdbf780045\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:20.057572 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.057288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0670c659-f08b-4285-98a4-eccdbf780045-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk\" (UID: \"0670c659-f08b-4285-98a4-eccdbf780045\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:20.057929 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.057911 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0670c659-f08b-4285-98a4-eccdbf780045-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk\" (UID: \"0670c659-f08b-4285-98a4-eccdbf780045\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:20.068496 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.068473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4vp\" (UniqueName: \"kubernetes.io/projected/0670c659-f08b-4285-98a4-eccdbf780045-kube-api-access-cr4vp\") pod \"payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk\" (UID: \"0670c659-f08b-4285-98a4-eccdbf780045\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:20.158402 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.158362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj\" (UID: \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:20.158585 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.158427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2kb8\" (UniqueName: \"kubernetes.io/projected/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-kube-api-access-n2kb8\") pod \"payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj\" (UID: \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:20.159159 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.159138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj\" (UID: \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:20.168090 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.168058 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2kb8\" (UniqueName: \"kubernetes.io/projected/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-kube-api-access-n2kb8\") pod \"payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj\" (UID: \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:20.170974 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.170952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:20.224501 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.224384 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:20.320470 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.320147 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk"] Jul 27 20:18:20.321970 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:18:20.321926 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0670c659_f08b_4285_98a4_eccdbf780045.slice/crio-97ee09300e4969bb96d13a2130d54575b289c68202f809e9cf6c43c530026711 WatchSource:0}: Error finding container 97ee09300e4969bb96d13a2130d54575b289c68202f809e9cf6c43c530026711: Status 404 returned error can't find the container with id 97ee09300e4969bb96d13a2130d54575b289c68202f809e9cf6c43c530026711 Jul 27 20:18:20.375856 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:20.375822 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj"] Jul 27 20:18:20.378933 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:18:20.378905 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0ced08_2acb_4dfd_bdf7_6051ef1ce422.slice/crio-a22d8c2d4f007f4d9dd11af95707f04cfd4e122d8e5aab67320a26b93b4cd305 WatchSource:0}: Error finding container a22d8c2d4f007f4d9dd11af95707f04cfd4e122d8e5aab67320a26b93b4cd305: Status 404 returned error can't find the container with id a22d8c2d4f007f4d9dd11af95707f04cfd4e122d8e5aab67320a26b93b4cd305 Jul 27 20:18:21.119069 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:21.119034 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" event={"ID":"0670c659-f08b-4285-98a4-eccdbf780045","Type":"ContainerStarted","Data":"2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b"} Jul 27 20:18:21.119554 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:21.119095 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:21.119554 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:21.119119 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" event={"ID":"0670c659-f08b-4285-98a4-eccdbf780045","Type":"ContainerStarted","Data":"97ee09300e4969bb96d13a2130d54575b289c68202f809e9cf6c43c530026711"} Jul 27 20:18:21.120674 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:21.120642 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" event={"ID":"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422","Type":"ContainerStarted","Data":"4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a"} Jul 27 20:18:21.120813 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:21.120691 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" event={"ID":"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422","Type":"ContainerStarted","Data":"a22d8c2d4f007f4d9dd11af95707f04cfd4e122d8e5aab67320a26b93b4cd305"} Jul 27 20:18:21.120813 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:21.120719 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:21.139953 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:21.139895 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" podStartSLOduration=2.139875796 podStartE2EDuration="2.139875796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:21.136543353 +0000 UTC m=+2305.749127956" watchObservedRunningTime="2026-07-27 20:18:21.139875796 +0000 UTC m=+2305.752460381" Jul 27 20:18:21.153340 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:21.153289 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" podStartSLOduration=2.153274509 podStartE2EDuration="2.153274509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:21.152055454 +0000 UTC m=+2305.764640038" watchObservedRunningTime="2026-07-27 20:18:21.153274509 +0000 UTC m=+2305.765859093" Jul 27 20:18:23.475970 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:23.475931 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:23.723117 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:23.723082 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:24.035363 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:24.035327 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:24.173231 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:24.173198 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:27.099706 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:27.099663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:27.252195 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:27.252162 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:27.356084 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:27.355998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:27.566308 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:27.566270 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:18:27.904818 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:27.904784 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:28.009947 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.009907 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:28.108383 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.108348 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:28.109175 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.109157 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg"] Jul 27 20:18:28.112662 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.112649 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:28.115163 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.115143 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-f44149\"" Jul 27 20:18:28.115319 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.115175 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-f44149-dockercfg-pnq9l\"" Jul 27 20:18:28.123021 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.122997 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg"] Jul 27 20:18:28.126181 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.126159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d2911-9880-4497-a887-d7c2b53a2bfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg\" (UID: \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:28.126276 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.126199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rkl\" (UniqueName: \"kubernetes.io/projected/8d5d2911-9880-4497-a887-d7c2b53a2bfc-kube-api-access-g2rkl\") pod \"payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg\" (UID: \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:28.138360 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.138335 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k"] Jul 27 20:18:28.146009 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.145988 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:28.150871 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.150851 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k"] Jul 27 20:18:28.227389 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.227358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d2911-9880-4497-a887-d7c2b53a2bfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg\" (UID: \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:28.227580 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.227415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvkz\" (UniqueName: \"kubernetes.io/projected/9ce38ce0-149f-446f-adc0-8088696ccac8-kube-api-access-ljvkz\") pod \"payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k\" (UID: \"9ce38ce0-149f-446f-adc0-8088696ccac8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:28.227580 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.227440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rkl\" (UniqueName: \"kubernetes.io/projected/8d5d2911-9880-4497-a887-d7c2b53a2bfc-kube-api-access-g2rkl\") pod \"payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg\" (UID: \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:28.227580 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.227472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce38ce0-149f-446f-adc0-8088696ccac8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k\" (UID: \"9ce38ce0-149f-446f-adc0-8088696ccac8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:28.228190 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.228168 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d2911-9880-4497-a887-d7c2b53a2bfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg\" (UID: \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:28.236214 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.236180 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rkl\" (UniqueName: \"kubernetes.io/projected/8d5d2911-9880-4497-a887-d7c2b53a2bfc-kube-api-access-g2rkl\") pod \"payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg\" (UID: \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:28.328680 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.328639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvkz\" (UniqueName: \"kubernetes.io/projected/9ce38ce0-149f-446f-adc0-8088696ccac8-kube-api-access-ljvkz\") pod \"payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k\" (UID: \"9ce38ce0-149f-446f-adc0-8088696ccac8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:28.328849 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.328695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce38ce0-149f-446f-adc0-8088696ccac8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k\" (UID: \"9ce38ce0-149f-446f-adc0-8088696ccac8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:28.329471 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.329452 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce38ce0-149f-446f-adc0-8088696ccac8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k\" (UID: \"9ce38ce0-149f-446f-adc0-8088696ccac8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:28.336949 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.336921 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvkz\" (UniqueName: \"kubernetes.io/projected/9ce38ce0-149f-446f-adc0-8088696ccac8-kube-api-access-ljvkz\") pod \"payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k\" (UID: \"9ce38ce0-149f-446f-adc0-8088696ccac8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:28.425200 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.425170 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:28.456478 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.456448 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:28.611059 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.607519 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg"] Jul 27 20:18:28.613050 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:18:28.613022 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5d2911_9880_4497_a887_d7c2b53a2bfc.slice/crio-986324f98d72411ae675a8a12f1a4ee400354d19c62fbda078ae2c7011ff005b WatchSource:0}: Error finding container 986324f98d72411ae675a8a12f1a4ee400354d19c62fbda078ae2c7011ff005b: Status 404 returned error can't find the container with id 986324f98d72411ae675a8a12f1a4ee400354d19c62fbda078ae2c7011ff005b Jul 27 20:18:28.634536 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:18:28.634505 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce38ce0_149f_446f_adc0_8088696ccac8.slice/crio-9441b94bc1afaf637f99cf472d2e800ac8909134c5045df4f72689ec5653dd66 WatchSource:0}: Error finding container 9441b94bc1afaf637f99cf472d2e800ac8909134c5045df4f72689ec5653dd66: Status 404 returned error can't find the container with id 9441b94bc1afaf637f99cf472d2e800ac8909134c5045df4f72689ec5653dd66 Jul 27 20:18:28.634894 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:28.634873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k"] Jul 27 20:18:29.166716 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:29.166666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" event={"ID":"9ce38ce0-149f-446f-adc0-8088696ccac8","Type":"ContainerStarted","Data":"b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c"} Jul 27 20:18:29.167143 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:29.166743 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:29.167143 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:29.166765 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" event={"ID":"9ce38ce0-149f-446f-adc0-8088696ccac8","Type":"ContainerStarted","Data":"9441b94bc1afaf637f99cf472d2e800ac8909134c5045df4f72689ec5653dd66"} Jul 27 20:18:29.168261 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:29.168240 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" event={"ID":"8d5d2911-9880-4497-a887-d7c2b53a2bfc","Type":"ContainerStarted","Data":"7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9"} Jul 27 20:18:29.168338 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:29.168280 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" event={"ID":"8d5d2911-9880-4497-a887-d7c2b53a2bfc","Type":"ContainerStarted","Data":"986324f98d72411ae675a8a12f1a4ee400354d19c62fbda078ae2c7011ff005b"} Jul 27 20:18:29.168338 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:29.168325 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:29.186378 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:29.186321 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" podStartSLOduration=1.18628683 podStartE2EDuration="1.18628683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:29.184236479 +0000 UTC m=+2313.796821078" watchObservedRunningTime="2026-07-27 20:18:29.18628683 +0000 UTC m=+2313.798871412" Jul 27 20:18:29.200694 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:29.200638 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" podStartSLOduration=1.200620047 podStartE2EDuration="1.200620047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:29.200091008 +0000 UTC m=+2313.812675592" watchObservedRunningTime="2026-07-27 20:18:29.200620047 +0000 UTC m=+2313.813204629" Jul 27 20:18:31.439115 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:31.439083 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:32.128481 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:32.128448 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:18:32.128675 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:32.128526 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:18:40.176045 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:40.176011 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:18:40.176513 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:40.176081 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:18:42.454719 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:42.454680 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:18:55.277581 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:55.277540 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:18:56.188060 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:56.188020 2573 scope.go:117] "RemoveContainer" containerID="3a98a147204a084c734f29505029700551d3d204e402e30321a435bf9789f250" Jul 27 20:18:58.938227 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:18:58.938176 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:19:13.066677 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:19:13.066636 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:19:16.960880 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:19:16.960840 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:19:48.447978 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:19:48.447941 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:20:10.543106 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:20:10.543069 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:20:10.551063 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:20:10.551036 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:20:10.633143 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:20:10.633101 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:20:29.344617 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:20:29.344537 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:20:43.616938 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:20:43.616905 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:20:55.931855 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:20:55.931816 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:21:02.180890 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:21:02.180846 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:21:14.342614 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:21:14.342571 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:21:37.372309 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:21:37.372258 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:21:56.236709 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:21:56.236631 2573 scope.go:117] "RemoveContainer" containerID="5c5dbc61cb7c8107420f821c937a71fc284bc78ac9c8313f34567100917177d8" Jul 27 20:21:56.527877 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:21:56.527787 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:22:23.833519 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:22:23.833483 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:22:26.667923 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:22:26.667884 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:22:42.331526 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:22:42.331471 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:22:56.258838 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:22:56.258806 2573 scope.go:117] "RemoveContainer" containerID="50e397098d4a1ec50dfb746f24a5cf7a8f56af3813c00a3fde582d3ab8a50866" Jul 27 20:23:03.234972 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:23:03.234942 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:23:21.637823 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:23:21.637736 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:23:23.570177 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:23:23.570139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:23:33.564038 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:23:33.564003 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:23:41.380232 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:23:41.380192 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:23:56.280255 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:23:56.280224 2573 scope.go:117] "RemoveContainer" containerID="74dbff40ec4b55e7d599d7708d80ff3aca03572014c16c1c82a1ed429c4d8133" Jul 27 20:23:56.364534 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:23:56.364500 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:24:00.568884 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:24:00.568848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:24:01.584566 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:24:01.584517 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:24:10.020280 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:24:10.020244 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:24:13.232352 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:24:13.232315 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:24:14.606974 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:24:14.606934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:24:46.662047 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:24:46.661976 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:24:51.028738 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:24:51.028700 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:25:26.033499 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:25:26.033459 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:25:45.835820 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:25:45.835779 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:25:51.256092 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:25:51.256038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:25:52.228042 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:25:52.228011 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:26:01.433889 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:26:01.433850 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:26:05.734746 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:26:05.734709 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:26:23.859424 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:26:23.859323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:26:27.648152 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:26:27.648118 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:27:00.336100 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:27:00.336059 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:27:18.633444 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:27:18.633407 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:27:27.742067 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:27:27.742023 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:27:39.929071 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:27:39.929035 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:27:47.394108 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:27:47.394022 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:27:48.537321 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:27:48.537261 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:27:54.039348 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:27:54.039307 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:28:06.787937 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:06.787895 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:07.097989 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:07.097910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:07.619251 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:07.619219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:10.482585 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:10.482553 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:10.578336 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:10.578275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:11.698830 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:11.698799 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:11.901829 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:11.901792 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:12.164476 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.164389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:12.264437 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.264402 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:12.272672 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.272641 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx"] Jul 27 20:28:12.275855 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.275839 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:12.278281 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.278263 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-555352\"" Jul 27 20:28:12.278388 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.278266 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-555352-dockercfg-bvjfg\"" Jul 27 20:28:12.293985 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.293957 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx"] Jul 27 20:28:12.308469 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.308443 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7"] Jul 27 20:28:12.311854 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.311832 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:12.323027 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.323002 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7"] Jul 27 20:28:12.438739 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.438711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwl9\" (UniqueName: \"kubernetes.io/projected/2a57175f-db33-483e-8a7d-b9afd2d003c9-kube-api-access-xvwl9\") pod \"payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx\" (UID: \"2a57175f-db33-483e-8a7d-b9afd2d003c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:12.438899 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.438746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7\" (UID: \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:12.438899 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.438859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ptg\" (UniqueName: \"kubernetes.io/projected/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-kube-api-access-c4ptg\") pod \"payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7\" (UID: \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:12.438978 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.438911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a57175f-db33-483e-8a7d-b9afd2d003c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx\" (UID: \"2a57175f-db33-483e-8a7d-b9afd2d003c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:12.539736 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.539705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ptg\" (UniqueName: \"kubernetes.io/projected/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-kube-api-access-c4ptg\") pod \"payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7\" (UID: \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:12.539933 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.539752 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a57175f-db33-483e-8a7d-b9afd2d003c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx\" (UID: \"2a57175f-db33-483e-8a7d-b9afd2d003c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:12.539933 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.539779 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwl9\" (UniqueName: \"kubernetes.io/projected/2a57175f-db33-483e-8a7d-b9afd2d003c9-kube-api-access-xvwl9\") pod \"payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx\" (UID: \"2a57175f-db33-483e-8a7d-b9afd2d003c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:12.539933 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.539802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7\" (UID: \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:12.540557 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.540539 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7\" (UID: \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:12.540611 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.540559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a57175f-db33-483e-8a7d-b9afd2d003c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx\" (UID: \"2a57175f-db33-483e-8a7d-b9afd2d003c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:12.547876 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.547844 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ptg\" (UniqueName: \"kubernetes.io/projected/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-kube-api-access-c4ptg\") pod \"payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7\" (UID: \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:12.548004 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.547882 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwl9\" (UniqueName: \"kubernetes.io/projected/2a57175f-db33-483e-8a7d-b9afd2d003c9-kube-api-access-xvwl9\") pod \"payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx\" (UID: \"2a57175f-db33-483e-8a7d-b9afd2d003c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:12.591740 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.591698 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:12.622716 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.622686 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:12.739609 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.739574 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx"] Jul 27 20:28:12.741533 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:28:12.741497 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a57175f_db33_483e_8a7d_b9afd2d003c9.slice/crio-5400fc20be37387650b9aa9a414a0d521478d112e914995180c76854fc276072 WatchSource:0}: Error finding container 5400fc20be37387650b9aa9a414a0d521478d112e914995180c76854fc276072: Status 404 returned error can't find the container with id 5400fc20be37387650b9aa9a414a0d521478d112e914995180c76854fc276072 Jul 27 20:28:12.778172 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:12.778138 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7"] Jul 27 20:28:12.780313 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:28:12.780276 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4d9f5e_ee14_4eaf_8271_bbd3d67756b2.slice/crio-6523e5025e9cdb856fdc36b35b2ea74650fb7f37cad457468131d52a7f2cc267 WatchSource:0}: Error finding container 6523e5025e9cdb856fdc36b35b2ea74650fb7f37cad457468131d52a7f2cc267: Status 404 returned error can't find the container with id 6523e5025e9cdb856fdc36b35b2ea74650fb7f37cad457468131d52a7f2cc267 Jul 27 20:28:13.361040 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:13.361008 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" event={"ID":"2a57175f-db33-483e-8a7d-b9afd2d003c9","Type":"ContainerStarted","Data":"2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb"} Jul 27 20:28:13.361260 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:13.361059 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" event={"ID":"2a57175f-db33-483e-8a7d-b9afd2d003c9","Type":"ContainerStarted","Data":"5400fc20be37387650b9aa9a414a0d521478d112e914995180c76854fc276072"} Jul 27 20:28:13.361260 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:13.361127 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:13.362636 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:13.362615 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" event={"ID":"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2","Type":"ContainerStarted","Data":"71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862"} Jul 27 20:28:13.362729 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:13.362649 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" event={"ID":"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2","Type":"ContainerStarted","Data":"6523e5025e9cdb856fdc36b35b2ea74650fb7f37cad457468131d52a7f2cc267"} Jul 27 20:28:13.362729 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:13.362675 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:13.376841 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:13.376790 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" podStartSLOduration=1.376774518 podStartE2EDuration="1.376774518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:28:13.376087543 +0000 UTC m=+2897.988672151" watchObservedRunningTime="2026-07-27 20:28:13.376774518 +0000 UTC m=+2897.989359105" Jul 27 20:28:13.393662 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:13.393608 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" podStartSLOduration=1.3935902470000001 podStartE2EDuration="1.393590247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:28:13.392873327 +0000 UTC m=+2898.005457934" watchObservedRunningTime="2026-07-27 20:28:13.393590247 +0000 UTC m=+2898.006174842" Jul 27 20:28:14.952214 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:14.952120 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:19.438063 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:19.438023 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:28:24.370500 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:24.370468 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:28:24.370884 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:24.370809 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:28:25.836311 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:25.836251 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:26.848046 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:26.848010 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:32.121458 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:32.121419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:32.218776 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:32.218739 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:33.839682 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:33.839635 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:34.271882 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.271847 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:34.371608 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.371573 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:34.418058 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.418017 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm"] Jul 27 20:28:34.421672 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.421656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:34.424249 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.424227 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1517d6\"" Jul 27 20:28:34.424401 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.424283 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1517d6-dockercfg-4twjs\"" Jul 27 20:28:34.429747 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.429723 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm"] Jul 27 20:28:34.442971 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.442940 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:34.452480 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.452447 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn"] Jul 27 20:28:34.455878 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.455859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:34.464592 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.464569 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn"] Jul 27 20:28:34.533611 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.533534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b627c-ca28-4a9b-8867-e6245f60aa49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn\" (UID: \"061b627c-ca28-4a9b-8867-e6245f60aa49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:34.533769 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.533574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x646k\" (UniqueName: \"kubernetes.io/projected/be2489c4-2314-40ef-96f7-38d3e805eef3-kube-api-access-x646k\") pod \"payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm\" (UID: \"be2489c4-2314-40ef-96f7-38d3e805eef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:34.533769 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.533733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2489c4-2314-40ef-96f7-38d3e805eef3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm\" (UID: \"be2489c4-2314-40ef-96f7-38d3e805eef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:34.533882 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.533820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw59p\" (UniqueName: \"kubernetes.io/projected/061b627c-ca28-4a9b-8867-e6245f60aa49-kube-api-access-vw59p\") pod \"payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn\" (UID: \"061b627c-ca28-4a9b-8867-e6245f60aa49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:34.634880 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.634843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2489c4-2314-40ef-96f7-38d3e805eef3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm\" (UID: \"be2489c4-2314-40ef-96f7-38d3e805eef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:34.635087 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.634895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw59p\" (UniqueName: \"kubernetes.io/projected/061b627c-ca28-4a9b-8867-e6245f60aa49-kube-api-access-vw59p\") pod \"payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn\" (UID: \"061b627c-ca28-4a9b-8867-e6245f60aa49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:34.635087 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.634937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b627c-ca28-4a9b-8867-e6245f60aa49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn\" (UID: \"061b627c-ca28-4a9b-8867-e6245f60aa49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:34.635087 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.634958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x646k\" (UniqueName: \"kubernetes.io/projected/be2489c4-2314-40ef-96f7-38d3e805eef3-kube-api-access-x646k\") pod \"payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm\" (UID: \"be2489c4-2314-40ef-96f7-38d3e805eef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:34.635625 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.635606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2489c4-2314-40ef-96f7-38d3e805eef3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm\" (UID: \"be2489c4-2314-40ef-96f7-38d3e805eef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:34.635730 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.635711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b627c-ca28-4a9b-8867-e6245f60aa49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn\" (UID: \"061b627c-ca28-4a9b-8867-e6245f60aa49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:34.643734 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.643708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw59p\" (UniqueName: \"kubernetes.io/projected/061b627c-ca28-4a9b-8867-e6245f60aa49-kube-api-access-vw59p\") pod \"payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn\" (UID: \"061b627c-ca28-4a9b-8867-e6245f60aa49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:34.643836 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.643818 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x646k\" (UniqueName: \"kubernetes.io/projected/be2489c4-2314-40ef-96f7-38d3e805eef3-kube-api-access-x646k\") pod \"payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm\" (UID: \"be2489c4-2314-40ef-96f7-38d3e805eef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:34.734742 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.734710 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:34.766909 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.766875 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:34.888703 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:34.888674 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm"] Jul 27 20:28:34.890531 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:28:34.890505 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2489c4_2314_40ef_96f7_38d3e805eef3.slice/crio-671c725bceb2ad6487b93d8c72318b8d60594dda101e905e04eb892df36dbd25 WatchSource:0}: Error finding container 671c725bceb2ad6487b93d8c72318b8d60594dda101e905e04eb892df36dbd25: Status 404 returned error can't find the container with id 671c725bceb2ad6487b93d8c72318b8d60594dda101e905e04eb892df36dbd25 Jul 27 20:28:35.121723 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:35.121700 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn"] Jul 27 20:28:35.123867 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:28:35.123843 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061b627c_ca28_4a9b_8867_e6245f60aa49.slice/crio-a95e0dfb3b8b6bd0533ecb623125d359336625bebee8c59d76a11904abab04e8 WatchSource:0}: Error finding container a95e0dfb3b8b6bd0533ecb623125d359336625bebee8c59d76a11904abab04e8: Status 404 returned error can't find the container with id a95e0dfb3b8b6bd0533ecb623125d359336625bebee8c59d76a11904abab04e8 Jul 27 20:28:35.488593 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:35.488565 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" event={"ID":"061b627c-ca28-4a9b-8867-e6245f60aa49","Type":"ContainerStarted","Data":"3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d"} Jul 27 20:28:35.488795 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:35.488622 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:35.488795 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:35.488646 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" event={"ID":"061b627c-ca28-4a9b-8867-e6245f60aa49","Type":"ContainerStarted","Data":"a95e0dfb3b8b6bd0533ecb623125d359336625bebee8c59d76a11904abab04e8"} Jul 27 20:28:35.490083 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:35.490059 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" event={"ID":"be2489c4-2314-40ef-96f7-38d3e805eef3","Type":"ContainerStarted","Data":"8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda"} Jul 27 20:28:35.490197 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:35.490101 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:35.490197 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:35.490116 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" event={"ID":"be2489c4-2314-40ef-96f7-38d3e805eef3","Type":"ContainerStarted","Data":"671c725bceb2ad6487b93d8c72318b8d60594dda101e905e04eb892df36dbd25"} Jul 27 20:28:35.507009 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:35.506955 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" podStartSLOduration=1.506935426 podStartE2EDuration="1.506935426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:28:35.505531922 +0000 UTC m=+2920.118116528" watchObservedRunningTime="2026-07-27 20:28:35.506935426 +0000 UTC m=+2920.119520012" Jul 27 20:28:35.522533 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:35.522472 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" podStartSLOduration=1.5224563450000002 podStartE2EDuration="1.522456345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:28:35.52160831 +0000 UTC m=+2920.134192909" watchObservedRunningTime="2026-07-27 20:28:35.522456345 +0000 UTC m=+2920.135040927" Jul 27 20:28:36.589450 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:36.589414 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:42.538172 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:42.538127 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:28:45.716267 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:45.716232 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:28:46.498217 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:46.498183 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:28:46.498415 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:46.498269 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:28:53.771741 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:28:53.771710 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn_061b627c-ca28-4a9b-8867-e6245f60aa49/payload-processing/0.log" Jul 27 20:29:00.162721 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:00.162682 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:29:04.235261 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:04.235224 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:29:07.541638 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:07.541584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:29:38.544350 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:38.544231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:29:38.644753 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:38.644691 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:29:43.634527 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:43.634482 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:29:53.871711 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:53.871682 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7_1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2/payload-processing/0.log" Jul 27 20:29:54.105650 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:54.105620 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn_061b627c-ca28-4a9b-8867-e6245f60aa49/payload-processing/0.log" Jul 27 20:29:54.388084 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:54.388044 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:29:54.681127 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:54.681087 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:29:55.483134 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:55.483096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:29:57.669782 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:57.669724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:29:57.768925 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:29:57.768893 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:00.135817 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.135782 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl"] Jul 27 20:30:00.139581 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.139561 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh"] Jul 27 20:30:00.139697 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.139676 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl" Jul 27 20:30:00.142281 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.142257 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:30:00.142898 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.142883 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn"] Jul 27 20:30:00.143004 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.142988 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh" Jul 27 20:30:00.146499 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.146477 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx"] Jul 27 20:30:00.146700 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.146589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn" Jul 27 20:30:00.151929 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.151905 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl"] Jul 27 20:30:00.152134 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.152021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx" Jul 27 20:30:00.153701 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.153676 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn"] Jul 27 20:30:00.154350 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.154332 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl" Jul 27 20:30:00.157140 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.157109 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx"] Jul 27 20:30:00.172389 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.172357 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh"] Jul 27 20:30:00.173649 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.173624 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh" Jul 27 20:30:00.212796 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.212766 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn" Jul 27 20:30:00.221964 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.221826 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx" Jul 27 20:30:00.266051 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.265046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:00.369967 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.369928 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl"] Jul 27 20:30:00.372047 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:30:00.371836 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aefe780_6f90_42e8_93d0_1766f89c6a9b.slice/crio-f98fd7c4093b54f5e92271a344d2f868ac5e612dfc3a8c36d9947e71cd67c3a5 WatchSource:0}: Error finding container f98fd7c4093b54f5e92271a344d2f868ac5e612dfc3a8c36d9947e71cd67c3a5: Status 404 returned error can't find the container with id f98fd7c4093b54f5e92271a344d2f868ac5e612dfc3a8c36d9947e71cd67c3a5 Jul 27 20:30:00.400754 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.400346 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh"] Jul 27 20:30:00.411053 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:30:00.410681 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75323609_4b13_424d_ab83_5585ed6f084d.slice/crio-9131deac12c90a0f8e723443d6315f78cbfa369003dff21c60921ccce44c084f WatchSource:0}: Error finding container 9131deac12c90a0f8e723443d6315f78cbfa369003dff21c60921ccce44c084f: Status 404 returned error can't find the container with id 9131deac12c90a0f8e723443d6315f78cbfa369003dff21c60921ccce44c084f Jul 27 20:30:00.442040 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.442009 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx"] Jul 27 20:30:00.449742 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:30:00.449713 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92913cdf_ef80_4445_8282_66ceafd42704.slice/crio-49f703c795cd1670a04a34b88fd2fb8b413d3f93313831e6a3f3879cecffcdb0 WatchSource:0}: Error finding container 49f703c795cd1670a04a34b88fd2fb8b413d3f93313831e6a3f3879cecffcdb0: Status 404 returned error can't find the container with id 49f703c795cd1670a04a34b88fd2fb8b413d3f93313831e6a3f3879cecffcdb0 Jul 27 20:30:00.459204 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.459179 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn"] Jul 27 20:30:00.464152 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:30:00.464123 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872cc807_7937_4545_b19f_58ce352284e1.slice/crio-18fbfee53d2f82addd188167749d4ed40d57740fab7ca2920399aff3fcf56751 WatchSource:0}: Error finding container 18fbfee53d2f82addd188167749d4ed40d57740fab7ca2920399aff3fcf56751: Status 404 returned error can't find the container with id 18fbfee53d2f82addd188167749d4ed40d57740fab7ca2920399aff3fcf56751 Jul 27 20:30:00.730425 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.730391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:00.836027 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.835993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:00.937044 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.937011 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf"] Jul 27 20:30:00.938526 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.938497 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:00.940878 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.940862 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:00.943427 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.943405 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-63bd67-dockercfg-gsb47\"" Jul 27 20:30:00.943524 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.943434 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-63bd67\"" Jul 27 20:30:00.950374 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.950353 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf"] Jul 27 20:30:00.981560 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.981422 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl" event={"ID":"7aefe780-6f90-42e8-93d0-1766f89c6a9b","Type":"ContainerStarted","Data":"6c57b7a3cb990a254324e0c4089436226cca6b6e3e3bf97caea13b9eb813ff31"} Jul 27 20:30:00.981764 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.981588 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl" event={"ID":"7aefe780-6f90-42e8-93d0-1766f89c6a9b","Type":"ContainerStarted","Data":"f98fd7c4093b54f5e92271a344d2f868ac5e612dfc3a8c36d9947e71cd67c3a5"} Jul 27 20:30:00.983061 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.983037 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn"] Jul 27 20:30:00.987480 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.987457 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn"] Jul 27 20:30:00.987618 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.987498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx" event={"ID":"92913cdf-ef80-4445-8282-66ceafd42704","Type":"ContainerStarted","Data":"2cb5ea93688da70a6a8cb0bc66f460692cab9893182d38d2dcc864b57b25c1bd"} Jul 27 20:30:00.987618 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.987526 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx" event={"ID":"92913cdf-ef80-4445-8282-66ceafd42704","Type":"ContainerStarted","Data":"49f703c795cd1670a04a34b88fd2fb8b413d3f93313831e6a3f3879cecffcdb0"} Jul 27 20:30:00.987618 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.987549 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh" event={"ID":"75323609-4b13-424d-ab83-5585ed6f084d","Type":"ContainerStarted","Data":"898977c7798e94d7cea04f07372b102310db7b0c1582bfd8d1eb9bbbc49476f7"} Jul 27 20:30:00.987618 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.987566 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:00.987618 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.987572 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh" event={"ID":"75323609-4b13-424d-ab83-5585ed6f084d","Type":"ContainerStarted","Data":"9131deac12c90a0f8e723443d6315f78cbfa369003dff21c60921ccce44c084f"} Jul 27 20:30:00.987860 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.987840 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn" event={"ID":"872cc807-7937-4545-b19f-58ce352284e1","Type":"ContainerStarted","Data":"1094868ee58dfbe9feae66cb186bfc14cfe0ee6cb5b25bc7690316613e11ebf9"} Jul 27 20:30:00.987924 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:00.987876 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn" event={"ID":"872cc807-7937-4545-b19f-58ce352284e1","Type":"ContainerStarted","Data":"18fbfee53d2f82addd188167749d4ed40d57740fab7ca2920399aff3fcf56751"} Jul 27 20:30:01.000095 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.000039 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl" podStartSLOduration=1.000017905 podStartE2EDuration="1.000017905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:00.996421505 +0000 UTC m=+3005.609006113" watchObservedRunningTime="2026-07-27 20:30:01.000017905 +0000 UTC m=+3005.612602525" Jul 27 20:30:01.010271 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.010224 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx" podStartSLOduration=1.010205442 podStartE2EDuration="1.010205442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:01.010127667 +0000 UTC m=+3005.622712270" watchObservedRunningTime="2026-07-27 20:30:01.010205442 +0000 UTC m=+3005.622790029" Jul 27 20:30:01.011055 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.011019 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875fa01b-25de-4627-8bfc-992f914720b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn\" (UID: \"875fa01b-25de-4627-8bfc-992f914720b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:01.011203 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.011181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjflf\" (UniqueName: \"kubernetes.io/projected/dea482a4-3e8c-489e-9c06-b32283469b02-kube-api-access-zjflf\") pod \"payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf\" (UID: \"dea482a4-3e8c-489e-9c06-b32283469b02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:01.011280 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.011226 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th58w\" (UniqueName: \"kubernetes.io/projected/875fa01b-25de-4627-8bfc-992f914720b2-kube-api-access-th58w\") pod \"payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn\" (UID: \"875fa01b-25de-4627-8bfc-992f914720b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:01.011280 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.011255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea482a4-3e8c-489e-9c06-b32283469b02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf\" (UID: \"dea482a4-3e8c-489e-9c06-b32283469b02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:01.042649 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.042597 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh" podStartSLOduration=1.04257751 podStartE2EDuration="1.04257751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:01.038281294 +0000 UTC m=+3005.650865904" watchObservedRunningTime="2026-07-27 20:30:01.04257751 +0000 UTC m=+3005.655162093" Jul 27 20:30:01.055186 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.055130 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn" podStartSLOduration=1.055112226 podStartE2EDuration="1.055112226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:01.052531404 +0000 UTC m=+3005.665115980" watchObservedRunningTime="2026-07-27 20:30:01.055112226 +0000 UTC m=+3005.667696809" Jul 27 20:30:01.112105 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.112056 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875fa01b-25de-4627-8bfc-992f914720b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn\" (UID: \"875fa01b-25de-4627-8bfc-992f914720b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:01.112279 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.112151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjflf\" (UniqueName: \"kubernetes.io/projected/dea482a4-3e8c-489e-9c06-b32283469b02-kube-api-access-zjflf\") pod \"payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf\" (UID: \"dea482a4-3e8c-489e-9c06-b32283469b02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:01.112279 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.112174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th58w\" (UniqueName: \"kubernetes.io/projected/875fa01b-25de-4627-8bfc-992f914720b2-kube-api-access-th58w\") pod \"payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn\" (UID: \"875fa01b-25de-4627-8bfc-992f914720b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:01.112279 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.112190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea482a4-3e8c-489e-9c06-b32283469b02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf\" (UID: \"dea482a4-3e8c-489e-9c06-b32283469b02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:01.112887 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.112866 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea482a4-3e8c-489e-9c06-b32283469b02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf\" (UID: \"dea482a4-3e8c-489e-9c06-b32283469b02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:01.112887 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.112877 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875fa01b-25de-4627-8bfc-992f914720b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn\" (UID: \"875fa01b-25de-4627-8bfc-992f914720b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:01.121560 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.121531 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjflf\" (UniqueName: \"kubernetes.io/projected/dea482a4-3e8c-489e-9c06-b32283469b02-kube-api-access-zjflf\") pod \"payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf\" (UID: \"dea482a4-3e8c-489e-9c06-b32283469b02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:01.121705 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.121540 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th58w\" (UniqueName: \"kubernetes.io/projected/875fa01b-25de-4627-8bfc-992f914720b2-kube-api-access-th58w\") pod \"payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn\" (UID: \"875fa01b-25de-4627-8bfc-992f914720b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:01.252868 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.252771 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:01.300271 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.300238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:01.413094 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.412952 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf"] Jul 27 20:30:01.475082 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.475054 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn"] Jul 27 20:30:01.477516 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:30:01.477472 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875fa01b_25de_4627_8bfc_992f914720b2.slice/crio-4dacca02a33ded2650af7983e38edc75941ec2fbbe0debe2da74287bb44e01b0 WatchSource:0}: Error finding container 4dacca02a33ded2650af7983e38edc75941ec2fbbe0debe2da74287bb44e01b0: Status 404 returned error can't find the container with id 4dacca02a33ded2650af7983e38edc75941ec2fbbe0debe2da74287bb44e01b0 Jul 27 20:30:01.995855 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.995817 2573 generic.go:356] "Generic (PLEG): container finished" podID="872cc807-7937-4545-b19f-58ce352284e1" containerID="1094868ee58dfbe9feae66cb186bfc14cfe0ee6cb5b25bc7690316613e11ebf9" exitCode=0 Jul 27 20:30:01.996063 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.995917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn" event={"ID":"872cc807-7937-4545-b19f-58ce352284e1","Type":"ContainerDied","Data":"1094868ee58dfbe9feae66cb186bfc14cfe0ee6cb5b25bc7690316613e11ebf9"} Jul 27 20:30:01.997591 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.997558 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" event={"ID":"875fa01b-25de-4627-8bfc-992f914720b2","Type":"ContainerStarted","Data":"05443a0b8f321a8658b9f2b9ca3dea03b68642f6257a7d4dce78053c5d672395"} Jul 27 20:30:01.997728 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.997612 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:01.997728 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.997637 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" event={"ID":"875fa01b-25de-4627-8bfc-992f914720b2","Type":"ContainerStarted","Data":"4dacca02a33ded2650af7983e38edc75941ec2fbbe0debe2da74287bb44e01b0"} Jul 27 20:30:01.999259 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.999184 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" event={"ID":"dea482a4-3e8c-489e-9c06-b32283469b02","Type":"ContainerStarted","Data":"e36a1cbd3962e6dd0c091ed7227fef34995dfa18950a27b9d4cc7a8f69e48259"} Jul 27 20:30:01.999259 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.999228 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" event={"ID":"dea482a4-3e8c-489e-9c06-b32283469b02","Type":"ContainerStarted","Data":"757ae1182f16400e2110d52b196c230bec62953bdc0f1c07a0ece3acb6b8eaea"} Jul 27 20:30:01.999472 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:01.999328 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:02.000860 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:02.000838 2573 generic.go:356] "Generic (PLEG): container finished" podID="7aefe780-6f90-42e8-93d0-1766f89c6a9b" containerID="6c57b7a3cb990a254324e0c4089436226cca6b6e3e3bf97caea13b9eb813ff31" exitCode=0 Jul 27 20:30:02.000961 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:02.000914 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl" event={"ID":"7aefe780-6f90-42e8-93d0-1766f89c6a9b","Type":"ContainerDied","Data":"6c57b7a3cb990a254324e0c4089436226cca6b6e3e3bf97caea13b9eb813ff31"} Jul 27 20:30:02.002419 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:02.002395 2573 generic.go:356] "Generic (PLEG): container finished" podID="92913cdf-ef80-4445-8282-66ceafd42704" containerID="2cb5ea93688da70a6a8cb0bc66f460692cab9893182d38d2dcc864b57b25c1bd" exitCode=0 Jul 27 20:30:02.002497 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:02.002471 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx" event={"ID":"92913cdf-ef80-4445-8282-66ceafd42704","Type":"ContainerDied","Data":"2cb5ea93688da70a6a8cb0bc66f460692cab9893182d38d2dcc864b57b25c1bd"} Jul 27 20:30:02.003813 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:02.003793 2573 generic.go:356] "Generic (PLEG): container finished" podID="75323609-4b13-424d-ab83-5585ed6f084d" containerID="898977c7798e94d7cea04f07372b102310db7b0c1582bfd8d1eb9bbbc49476f7" exitCode=0 Jul 27 20:30:02.003892 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:02.003837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh" event={"ID":"75323609-4b13-424d-ab83-5585ed6f084d","Type":"ContainerDied","Data":"898977c7798e94d7cea04f07372b102310db7b0c1582bfd8d1eb9bbbc49476f7"} Jul 27 20:30:02.128181 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:02.128127 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" podStartSLOduration=2.128111584 podStartE2EDuration="2.128111584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:02.125345493 +0000 UTC m=+3006.737930095" watchObservedRunningTime="2026-07-27 20:30:02.128111584 +0000 UTC m=+3006.740696166" Jul 27 20:30:02.170082 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:02.170043 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:02.202974 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:02.202911 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" podStartSLOduration=2.202887129 podStartE2EDuration="2.202887129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:02.198255397 +0000 UTC m=+3006.810840015" watchObservedRunningTime="2026-07-27 20:30:02.202887129 +0000 UTC m=+3006.815471714" Jul 27 20:30:03.162950 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:03.162915 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl" Jul 27 20:30:03.223028 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:03.222996 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn" Jul 27 20:30:03.261220 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:03.261194 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh" Jul 27 20:30:03.265226 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:03.265155 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx" Jul 27 20:30:04.020045 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.020016 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl" Jul 27 20:30:04.020248 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.020014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl" event={"ID":"7aefe780-6f90-42e8-93d0-1766f89c6a9b","Type":"ContainerDied","Data":"f98fd7c4093b54f5e92271a344d2f868ac5e612dfc3a8c36d9947e71cd67c3a5"} Jul 27 20:30:04.020248 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.020163 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f98fd7c4093b54f5e92271a344d2f868ac5e612dfc3a8c36d9947e71cd67c3a5" Jul 27 20:30:04.021477 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.021441 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx" event={"ID":"92913cdf-ef80-4445-8282-66ceafd42704","Type":"ContainerDied","Data":"49f703c795cd1670a04a34b88fd2fb8b413d3f93313831e6a3f3879cecffcdb0"} Jul 27 20:30:04.021618 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.021490 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49f703c795cd1670a04a34b88fd2fb8b413d3f93313831e6a3f3879cecffcdb0" Jul 27 20:30:04.021618 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.021454 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx" Jul 27 20:30:04.023029 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.023001 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh" event={"ID":"75323609-4b13-424d-ab83-5585ed6f084d","Type":"ContainerDied","Data":"9131deac12c90a0f8e723443d6315f78cbfa369003dff21c60921ccce44c084f"} Jul 27 20:30:04.023166 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.023049 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9131deac12c90a0f8e723443d6315f78cbfa369003dff21c60921ccce44c084f" Jul 27 20:30:04.023166 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.023065 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh" Jul 27 20:30:04.024543 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.024518 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn" Jul 27 20:30:04.024660 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.024569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn" event={"ID":"872cc807-7937-4545-b19f-58ce352284e1","Type":"ContainerDied","Data":"18fbfee53d2f82addd188167749d4ed40d57740fab7ca2920399aff3fcf56751"} Jul 27 20:30:04.024660 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:04.024613 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18fbfee53d2f82addd188167749d4ed40d57740fab7ca2920399aff3fcf56751" Jul 27 20:30:08.427805 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:08.427759 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:10.529410 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:10.529378 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:30:13.013625 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:13.013600 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:13.013998 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:13.013672 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:18.109869 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.109833 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l"] Jul 27 20:30:18.110289 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110260 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aefe780-6f90-42e8-93d0-1766f89c6a9b" containerName="cleanup" Jul 27 20:30:18.110289 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110271 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aefe780-6f90-42e8-93d0-1766f89c6a9b" containerName="cleanup" Jul 27 20:30:18.110289 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110307 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92913cdf-ef80-4445-8282-66ceafd42704" containerName="cleanup" Jul 27 20:30:18.110289 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110313 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="92913cdf-ef80-4445-8282-66ceafd42704" containerName="cleanup" Jul 27 20:30:18.110289 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110320 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75323609-4b13-424d-ab83-5585ed6f084d" containerName="cleanup" Jul 27 20:30:18.110512 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110326 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="75323609-4b13-424d-ab83-5585ed6f084d" containerName="cleanup" Jul 27 20:30:18.110512 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110337 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872cc807-7937-4545-b19f-58ce352284e1" containerName="cleanup" Jul 27 20:30:18.110512 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110342 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="872cc807-7937-4545-b19f-58ce352284e1" containerName="cleanup" Jul 27 20:30:18.110512 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110399 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="872cc807-7937-4545-b19f-58ce352284e1" containerName="cleanup" Jul 27 20:30:18.110512 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110405 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aefe780-6f90-42e8-93d0-1766f89c6a9b" containerName="cleanup" Jul 27 20:30:18.110512 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110411 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="92913cdf-ef80-4445-8282-66ceafd42704" containerName="cleanup" Jul 27 20:30:18.110512 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.110420 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="75323609-4b13-424d-ab83-5585ed6f084d" containerName="cleanup" Jul 27 20:30:18.113737 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.113720 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" Jul 27 20:30:18.116214 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.116180 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:30:18.116366 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.116226 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:30:18.120929 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.120903 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l"] Jul 27 20:30:18.164683 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.164649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44515191-d256-4470-a20f-6ffd29e304e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l\" (UID: \"44515191-d256-4470-a20f-6ffd29e304e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" Jul 27 20:30:18.265822 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.265783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44515191-d256-4470-a20f-6ffd29e304e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l\" (UID: \"44515191-d256-4470-a20f-6ffd29e304e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" Jul 27 20:30:18.266481 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.266460 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44515191-d256-4470-a20f-6ffd29e304e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l\" (UID: \"44515191-d256-4470-a20f-6ffd29e304e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" Jul 27 20:30:18.425228 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.425150 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" Jul 27 20:30:18.777156 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:18.777120 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l"] Jul 27 20:30:18.779713 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:30:18.779670 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44515191_d256_4470_a20f_6ffd29e304e9.slice/crio-f6931c7b8b2f433c5af6c80f0945a9f0484051740eb20a701990b1c5aa835ce8 WatchSource:0}: Error finding container f6931c7b8b2f433c5af6c80f0945a9f0484051740eb20a701990b1c5aa835ce8: Status 404 returned error can't find the container with id f6931c7b8b2f433c5af6c80f0945a9f0484051740eb20a701990b1c5aa835ce8 Jul 27 20:30:19.111732 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:19.111704 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" event={"ID":"44515191-d256-4470-a20f-6ffd29e304e9","Type":"ContainerStarted","Data":"f84ade43b0ad89915626ee517a78c3609078f7a59388caea8d0b9a006193a8e9"} Jul 27 20:30:19.112091 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:19.111754 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" event={"ID":"44515191-d256-4470-a20f-6ffd29e304e9","Type":"ContainerStarted","Data":"f6931c7b8b2f433c5af6c80f0945a9f0484051740eb20a701990b1c5aa835ce8"} Jul 27 20:30:19.130522 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:19.130462 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" podStartSLOduration=1.130446608 podStartE2EDuration="1.130446608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:19.128119682 +0000 UTC m=+3023.740704266" watchObservedRunningTime="2026-07-27 20:30:19.130446608 +0000 UTC m=+3023.743031191" Jul 27 20:30:20.119323 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:20.119259 2573 generic.go:356] "Generic (PLEG): container finished" podID="44515191-d256-4470-a20f-6ffd29e304e9" containerID="f84ade43b0ad89915626ee517a78c3609078f7a59388caea8d0b9a006193a8e9" exitCode=0 Jul 27 20:30:20.119715 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:20.119349 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" event={"ID":"44515191-d256-4470-a20f-6ffd29e304e9","Type":"ContainerDied","Data":"f84ade43b0ad89915626ee517a78c3609078f7a59388caea8d0b9a006193a8e9"} Jul 27 20:30:21.263699 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:21.263671 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" Jul 27 20:30:21.397374 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:21.397274 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44515191-d256-4470-a20f-6ffd29e304e9-maas-api-service-ca\") pod \"44515191-d256-4470-a20f-6ffd29e304e9\" (UID: \"44515191-d256-4470-a20f-6ffd29e304e9\") " Jul 27 20:30:21.397673 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:21.397645 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44515191-d256-4470-a20f-6ffd29e304e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44515191-d256-4470-a20f-6ffd29e304e9" (UID: "44515191-d256-4470-a20f-6ffd29e304e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:30:21.497977 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:21.497947 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44515191-d256-4470-a20f-6ffd29e304e9-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:30:22.133372 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:22.133281 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" Jul 27 20:30:22.133372 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:22.133318 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l" event={"ID":"44515191-d256-4470-a20f-6ffd29e304e9","Type":"ContainerDied","Data":"f6931c7b8b2f433c5af6c80f0945a9f0484051740eb20a701990b1c5aa835ce8"} Jul 27 20:30:22.133372 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:22.133364 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6931c7b8b2f433c5af6c80f0945a9f0484051740eb20a701990b1c5aa835ce8" Jul 27 20:30:28.903344 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:28.903287 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:28.932779 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:28.932745 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn"] Jul 27 20:30:28.933000 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:28.932966 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" podUID="875fa01b-25de-4627-8bfc-992f914720b2" containerName="payload-processing" containerID="cri-o://05443a0b8f321a8658b9f2b9ca3dea03b68642f6257a7d4dce78053c5d672395" gracePeriod=30 Jul 27 20:30:28.942221 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:28.941489 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf"] Jul 27 20:30:28.942438 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:28.942252 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" podUID="dea482a4-3e8c-489e-9c06-b32283469b02" containerName="payload-pre-processing" containerID="cri-o://e36a1cbd3962e6dd0c091ed7227fef34995dfa18950a27b9d4cc7a8f69e48259" gracePeriod=30 Jul 27 20:30:28.979164 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:28.979107 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:29.059327 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.059276 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:29.177645 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.177554 2573 generic.go:356] "Generic (PLEG): container finished" podID="875fa01b-25de-4627-8bfc-992f914720b2" containerID="05443a0b8f321a8658b9f2b9ca3dea03b68642f6257a7d4dce78053c5d672395" exitCode=0 Jul 27 20:30:29.177645 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.177619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" event={"ID":"875fa01b-25de-4627-8bfc-992f914720b2","Type":"ContainerDied","Data":"05443a0b8f321a8658b9f2b9ca3dea03b68642f6257a7d4dce78053c5d672395"} Jul 27 20:30:29.179610 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.179580 2573 generic.go:356] "Generic (PLEG): container finished" podID="dea482a4-3e8c-489e-9c06-b32283469b02" containerID="e36a1cbd3962e6dd0c091ed7227fef34995dfa18950a27b9d4cc7a8f69e48259" exitCode=0 Jul 27 20:30:29.179794 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.179625 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" event={"ID":"dea482a4-3e8c-489e-9c06-b32283469b02","Type":"ContainerDied","Data":"e36a1cbd3962e6dd0c091ed7227fef34995dfa18950a27b9d4cc7a8f69e48259"} Jul 27 20:30:29.231803 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.231776 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:29.234757 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.234736 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:29.373176 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.373140 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea482a4-3e8c-489e-9c06-b32283469b02-plugins-config-volume\") pod \"dea482a4-3e8c-489e-9c06-b32283469b02\" (UID: \"dea482a4-3e8c-489e-9c06-b32283469b02\") " Jul 27 20:30:29.373176 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.373184 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjflf\" (UniqueName: \"kubernetes.io/projected/dea482a4-3e8c-489e-9c06-b32283469b02-kube-api-access-zjflf\") pod \"dea482a4-3e8c-489e-9c06-b32283469b02\" (UID: \"dea482a4-3e8c-489e-9c06-b32283469b02\") " Jul 27 20:30:29.373473 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.373234 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th58w\" (UniqueName: \"kubernetes.io/projected/875fa01b-25de-4627-8bfc-992f914720b2-kube-api-access-th58w\") pod \"875fa01b-25de-4627-8bfc-992f914720b2\" (UID: \"875fa01b-25de-4627-8bfc-992f914720b2\") " Jul 27 20:30:29.373473 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.373328 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875fa01b-25de-4627-8bfc-992f914720b2-plugins-config-volume\") pod \"875fa01b-25de-4627-8bfc-992f914720b2\" (UID: \"875fa01b-25de-4627-8bfc-992f914720b2\") " Jul 27 20:30:29.373660 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.373621 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea482a4-3e8c-489e-9c06-b32283469b02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dea482a4-3e8c-489e-9c06-b32283469b02" (UID: "dea482a4-3e8c-489e-9c06-b32283469b02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:30:29.373774 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.373715 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875fa01b-25de-4627-8bfc-992f914720b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "875fa01b-25de-4627-8bfc-992f914720b2" (UID: "875fa01b-25de-4627-8bfc-992f914720b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:30:29.375602 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.375582 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875fa01b-25de-4627-8bfc-992f914720b2-kube-api-access-th58w" (OuterVolumeSpecName: "kube-api-access-th58w") pod "875fa01b-25de-4627-8bfc-992f914720b2" (UID: "875fa01b-25de-4627-8bfc-992f914720b2"). InnerVolumeSpecName "kube-api-access-th58w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:30:29.375602 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.375586 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea482a4-3e8c-489e-9c06-b32283469b02-kube-api-access-zjflf" (OuterVolumeSpecName: "kube-api-access-zjflf") pod "dea482a4-3e8c-489e-9c06-b32283469b02" (UID: "dea482a4-3e8c-489e-9c06-b32283469b02"). InnerVolumeSpecName "kube-api-access-zjflf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:30:29.474949 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.474908 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875fa01b-25de-4627-8bfc-992f914720b2-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:30:29.474949 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.474943 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea482a4-3e8c-489e-9c06-b32283469b02-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:30:29.474949 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.474958 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjflf\" (UniqueName: \"kubernetes.io/projected/dea482a4-3e8c-489e-9c06-b32283469b02-kube-api-access-zjflf\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:30:29.475198 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:29.474970 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th58w\" (UniqueName: \"kubernetes.io/projected/875fa01b-25de-4627-8bfc-992f914720b2-kube-api-access-th58w\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:30:30.186795 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.186770 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" Jul 27 20:30:30.187253 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.186798 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn" event={"ID":"875fa01b-25de-4627-8bfc-992f914720b2","Type":"ContainerDied","Data":"4dacca02a33ded2650af7983e38edc75941ec2fbbe0debe2da74287bb44e01b0"} Jul 27 20:30:30.187253 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.186862 2573 scope.go:117] "RemoveContainer" containerID="05443a0b8f321a8658b9f2b9ca3dea03b68642f6257a7d4dce78053c5d672395" Jul 27 20:30:30.188249 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.188230 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" Jul 27 20:30:30.188437 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.188276 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf" event={"ID":"dea482a4-3e8c-489e-9c06-b32283469b02","Type":"ContainerDied","Data":"757ae1182f16400e2110d52b196c230bec62953bdc0f1c07a0ece3acb6b8eaea"} Jul 27 20:30:30.196366 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.196346 2573 scope.go:117] "RemoveContainer" containerID="e36a1cbd3962e6dd0c091ed7227fef34995dfa18950a27b9d4cc7a8f69e48259" Jul 27 20:30:30.206278 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.206251 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf"] Jul 27 20:30:30.209915 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.209895 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-63bd67-77477f577b-vx5nf"] Jul 27 20:30:30.219582 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.219562 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn"] Jul 27 20:30:30.221956 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:30.221939 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-63bd67-58c9bd5f5b-rwshn"] Jul 27 20:30:31.875887 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:31.875854 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="875fa01b-25de-4627-8bfc-992f914720b2" path="/var/lib/kubelet/pods/875fa01b-25de-4627-8bfc-992f914720b2/volumes" Jul 27 20:30:31.876252 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:31.876209 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dea482a4-3e8c-489e-9c06-b32283469b02" path="/var/lib/kubelet/pods/dea482a4-3e8c-489e-9c06-b32283469b02/volumes" Jul 27 20:30:33.367928 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:33.367891 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l"] Jul 27 20:30:33.370337 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:33.370310 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-63bd67-vpk8l"] Jul 27 20:30:33.372553 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:33.372524 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:33.742877 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:33.742835 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:33.876636 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:33.876602 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44515191-d256-4470-a20f-6ffd29e304e9" path="/var/lib/kubelet/pods/44515191-d256-4470-a20f-6ffd29e304e9/volumes" Jul 27 20:30:34.037241 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:34.037158 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:30:40.011227 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.011183 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:40.111199 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.111157 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:40.234494 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.234460 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ltpch"] Jul 27 20:30:40.234880 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.234868 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44515191-d256-4470-a20f-6ffd29e304e9" containerName="revoke-keys" Jul 27 20:30:40.234936 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.234881 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="44515191-d256-4470-a20f-6ffd29e304e9" containerName="revoke-keys" Jul 27 20:30:40.234936 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.234903 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea482a4-3e8c-489e-9c06-b32283469b02" containerName="payload-pre-processing" Jul 27 20:30:40.234936 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.234908 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea482a4-3e8c-489e-9c06-b32283469b02" containerName="payload-pre-processing" Jul 27 20:30:40.234936 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.234915 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875fa01b-25de-4627-8bfc-992f914720b2" containerName="payload-processing" Jul 27 20:30:40.234936 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.234922 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="875fa01b-25de-4627-8bfc-992f914720b2" containerName="payload-processing" Jul 27 20:30:40.235097 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.234993 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dea482a4-3e8c-489e-9c06-b32283469b02" containerName="payload-pre-processing" Jul 27 20:30:40.235097 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.235000 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="875fa01b-25de-4627-8bfc-992f914720b2" containerName="payload-processing" Jul 27 20:30:40.235097 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.235008 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="44515191-d256-4470-a20f-6ffd29e304e9" containerName="revoke-keys" Jul 27 20:30:40.238243 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.238228 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:40.242505 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.242483 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-bm96v\"" Jul 27 20:30:40.267432 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.267351 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ltpch"] Jul 27 20:30:40.352667 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.352627 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:40.376613 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.376574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zp4\" (UniqueName: \"kubernetes.io/projected/684a9029-760a-4d08-ba85-b5472b3e4d0c-kube-api-access-52zp4\") pod \"kuadrant-operator-catalog-ltpch\" (UID: \"684a9029-760a-4d08-ba85-b5472b3e4d0c\") " pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:40.478001 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.477959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zp4\" (UniqueName: \"kubernetes.io/projected/684a9029-760a-4d08-ba85-b5472b3e4d0c-kube-api-access-52zp4\") pod \"kuadrant-operator-catalog-ltpch\" (UID: \"684a9029-760a-4d08-ba85-b5472b3e4d0c\") " pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:40.487099 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.487066 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zp4\" (UniqueName: \"kubernetes.io/projected/684a9029-760a-4d08-ba85-b5472b3e4d0c-kube-api-access-52zp4\") pod \"kuadrant-operator-catalog-ltpch\" (UID: \"684a9029-760a-4d08-ba85-b5472b3e4d0c\") " pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:40.511654 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.511614 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:30:40.548644 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.548556 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:40.680672 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.680641 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ltpch"] Jul 27 20:30:40.682308 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:30:40.682263 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684a9029_760a_4d08_ba85_b5472b3e4d0c.slice/crio-c58beb29ab90ac67642d5bd7a8623a4c71b2783b33076e43e9d6fc3edc175459 WatchSource:0}: Error finding container c58beb29ab90ac67642d5bd7a8623a4c71b2783b33076e43e9d6fc3edc175459: Status 404 returned error can't find the container with id c58beb29ab90ac67642d5bd7a8623a4c71b2783b33076e43e9d6fc3edc175459 Jul 27 20:30:40.683542 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:40.683525 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 20:30:41.257882 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:41.257846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" event={"ID":"684a9029-760a-4d08-ba85-b5472b3e4d0c","Type":"ContainerStarted","Data":"c58beb29ab90ac67642d5bd7a8623a4c71b2783b33076e43e9d6fc3edc175459"} Jul 27 20:30:43.272501 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:43.272468 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" event={"ID":"684a9029-760a-4d08-ba85-b5472b3e4d0c","Type":"ContainerStarted","Data":"ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22"} Jul 27 20:30:43.289056 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:43.289014 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" podStartSLOduration=1.5985469719999998 podStartE2EDuration="3.288999913s" totalImagesPullingTime="1.690452941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:43.287616166 +0000 UTC m=+3047.900200779" watchObservedRunningTime="2026-07-27 20:30:43.288999913 +0000 UTC m=+3047.901584496" Jul 27 20:30:43.735538 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:43.735501 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8"] Jul 27 20:30:43.739199 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:43.739184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" Jul 27 20:30:43.741747 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:43.741727 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:30:43.741879 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:43.741730 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:30:43.746585 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:43.746560 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8"] Jul 27 20:30:43.908381 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:43.908342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3249c41-4da0-44b5-88b7-491d1fabc275-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8\" (UID: \"c3249c41-4da0-44b5-88b7-491d1fabc275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" Jul 27 20:30:44.009820 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:44.009708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3249c41-4da0-44b5-88b7-491d1fabc275-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8\" (UID: \"c3249c41-4da0-44b5-88b7-491d1fabc275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" Jul 27 20:30:44.010563 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:44.010542 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3249c41-4da0-44b5-88b7-491d1fabc275-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8\" (UID: \"c3249c41-4da0-44b5-88b7-491d1fabc275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" Jul 27 20:30:44.050605 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:44.050573 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" Jul 27 20:30:44.188751 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:44.188722 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8"] Jul 27 20:30:44.190434 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:30:44.190409 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3249c41_4da0_44b5_88b7_491d1fabc275.slice/crio-cf8b3caf77482bb0c8a815afa30a179cb5b6addd71f42431755a89262d03decb WatchSource:0}: Error finding container cf8b3caf77482bb0c8a815afa30a179cb5b6addd71f42431755a89262d03decb: Status 404 returned error can't find the container with id cf8b3caf77482bb0c8a815afa30a179cb5b6addd71f42431755a89262d03decb Jul 27 20:30:44.286710 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:44.286642 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" event={"ID":"c3249c41-4da0-44b5-88b7-491d1fabc275","Type":"ContainerStarted","Data":"cf8b3caf77482bb0c8a815afa30a179cb5b6addd71f42431755a89262d03decb"} Jul 27 20:30:45.294720 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:45.294684 2573 generic.go:356] "Generic (PLEG): container finished" podID="c3249c41-4da0-44b5-88b7-491d1fabc275" containerID="16839fada0482399eb8c2f329b01d6d5f9224bf39c417d87783827b2153261e4" exitCode=0 Jul 27 20:30:45.295114 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:45.294770 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" event={"ID":"c3249c41-4da0-44b5-88b7-491d1fabc275","Type":"ContainerDied","Data":"16839fada0482399eb8c2f329b01d6d5f9224bf39c417d87783827b2153261e4"} Jul 27 20:30:46.437763 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:46.437739 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" Jul 27 20:30:46.536554 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:46.536521 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3249c41-4da0-44b5-88b7-491d1fabc275-maas-api-service-ca\") pod \"c3249c41-4da0-44b5-88b7-491d1fabc275\" (UID: \"c3249c41-4da0-44b5-88b7-491d1fabc275\") " Jul 27 20:30:46.536886 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:46.536864 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3249c41-4da0-44b5-88b7-491d1fabc275-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3249c41-4da0-44b5-88b7-491d1fabc275" (UID: "c3249c41-4da0-44b5-88b7-491d1fabc275"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:30:46.637955 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:46.637879 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3249c41-4da0-44b5-88b7-491d1fabc275-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:30:47.308188 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:47.308158 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" Jul 27 20:30:47.308398 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:47.308151 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8" event={"ID":"c3249c41-4da0-44b5-88b7-491d1fabc275","Type":"ContainerDied","Data":"cf8b3caf77482bb0c8a815afa30a179cb5b6addd71f42431755a89262d03decb"} Jul 27 20:30:47.308398 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:47.308282 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf8b3caf77482bb0c8a815afa30a179cb5b6addd71f42431755a89262d03decb" Jul 27 20:30:50.549635 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:50.549598 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:50.550087 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:50.549665 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:50.572042 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:50.572015 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:51.354287 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:51.354258 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:51.390558 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:51.390522 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ltpch"] Jul 27 20:30:53.344244 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:53.344193 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" podUID="684a9029-760a-4d08-ba85-b5472b3e4d0c" containerName="registry-server" containerID="cri-o://ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22" gracePeriod=2 Jul 27 20:30:53.621226 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:53.621155 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:53.698966 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:53.698926 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52zp4\" (UniqueName: \"kubernetes.io/projected/684a9029-760a-4d08-ba85-b5472b3e4d0c-kube-api-access-52zp4\") pod \"684a9029-760a-4d08-ba85-b5472b3e4d0c\" (UID: \"684a9029-760a-4d08-ba85-b5472b3e4d0c\") " Jul 27 20:30:53.701115 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:53.701092 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684a9029-760a-4d08-ba85-b5472b3e4d0c-kube-api-access-52zp4" (OuterVolumeSpecName: "kube-api-access-52zp4") pod "684a9029-760a-4d08-ba85-b5472b3e4d0c" (UID: "684a9029-760a-4d08-ba85-b5472b3e4d0c"). InnerVolumeSpecName "kube-api-access-52zp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:30:53.800111 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:53.800079 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52zp4\" (UniqueName: \"kubernetes.io/projected/684a9029-760a-4d08-ba85-b5472b3e4d0c-kube-api-access-52zp4\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:30:54.351635 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:54.351598 2573 generic.go:356] "Generic (PLEG): container finished" podID="684a9029-760a-4d08-ba85-b5472b3e4d0c" containerID="ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22" exitCode=0 Jul 27 20:30:54.351635 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:54.351646 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" event={"ID":"684a9029-760a-4d08-ba85-b5472b3e4d0c","Type":"ContainerDied","Data":"ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22"} Jul 27 20:30:54.352185 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:54.351685 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" event={"ID":"684a9029-760a-4d08-ba85-b5472b3e4d0c","Type":"ContainerDied","Data":"c58beb29ab90ac67642d5bd7a8623a4c71b2783b33076e43e9d6fc3edc175459"} Jul 27 20:30:54.352185 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:54.351705 2573 scope.go:117] "RemoveContainer" containerID="ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22" Jul 27 20:30:54.352185 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:54.351647 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ltpch" Jul 27 20:30:54.364100 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:54.364079 2573 scope.go:117] "RemoveContainer" containerID="ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22" Jul 27 20:30:54.364366 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:30:54.364344 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22\": container with ID starting with ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22 not found: ID does not exist" containerID="ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22" Jul 27 20:30:54.364430 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:54.364375 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22"} err="failed to get container status \"ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22\": rpc error: code = NotFound desc = could not find container \"ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22\": container with ID starting with ca16b29b2561a36a10da828cbb2f0c5d7c168194a28718e8fbef610666d17a22 not found: ID does not exist" Jul 27 20:30:54.369055 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:54.369032 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ltpch"] Jul 27 20:30:54.371825 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:54.371806 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ltpch"] Jul 27 20:30:55.881007 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:30:55.880970 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="684a9029-760a-4d08-ba85-b5472b3e4d0c" path="/var/lib/kubelet/pods/684a9029-760a-4d08-ba85-b5472b3e4d0c/volumes" Jul 27 20:31:01.123550 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:01.123503 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:31:04.312086 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.312051 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx"] Jul 27 20:31:04.314606 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.314582 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-555352-29753070-7rfrx"] Jul 27 20:31:04.318591 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.318566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:31:04.353328 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.353281 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7"] Jul 27 20:31:04.353607 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.353579 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" podUID="1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2" containerName="payload-processing" containerID="cri-o://71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862" gracePeriod=30 Jul 27 20:31:04.362144 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.362115 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx"] Jul 27 20:31:04.362417 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.362386 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" podUID="2a57175f-db33-483e-8a7d-b9afd2d003c9" containerName="payload-pre-processing" containerID="cri-o://2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb" gracePeriod=30 Jul 27 20:31:04.369163 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.369119 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" podUID="1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.75:9004: connect: connection refused" Jul 27 20:31:04.369337 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.369119 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" podUID="2a57175f-db33-483e-8a7d-b9afd2d003c9" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.74:9004: connect: connection refused" Jul 27 20:31:04.390886 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.390851 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:31:04.488407 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.488372 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:31:04.642402 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.642380 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:31:04.645330 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.645286 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:31:04.692616 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.692589 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a57175f-db33-483e-8a7d-b9afd2d003c9-plugins-config-volume\") pod \"2a57175f-db33-483e-8a7d-b9afd2d003c9\" (UID: \"2a57175f-db33-483e-8a7d-b9afd2d003c9\") " Jul 27 20:31:04.692811 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.692648 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-plugins-config-volume\") pod \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\" (UID: \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\") " Jul 27 20:31:04.692811 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.692697 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4ptg\" (UniqueName: \"kubernetes.io/projected/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-kube-api-access-c4ptg\") pod \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\" (UID: \"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2\") " Jul 27 20:31:04.692811 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.692723 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvwl9\" (UniqueName: \"kubernetes.io/projected/2a57175f-db33-483e-8a7d-b9afd2d003c9-kube-api-access-xvwl9\") pod \"2a57175f-db33-483e-8a7d-b9afd2d003c9\" (UID: \"2a57175f-db33-483e-8a7d-b9afd2d003c9\") " Jul 27 20:31:04.693021 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.692995 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a57175f-db33-483e-8a7d-b9afd2d003c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a57175f-db33-483e-8a7d-b9afd2d003c9" (UID: "2a57175f-db33-483e-8a7d-b9afd2d003c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:31:04.693143 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.693090 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2" (UID: "1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:31:04.695138 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.695116 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a57175f-db33-483e-8a7d-b9afd2d003c9-kube-api-access-xvwl9" (OuterVolumeSpecName: "kube-api-access-xvwl9") pod "2a57175f-db33-483e-8a7d-b9afd2d003c9" (UID: "2a57175f-db33-483e-8a7d-b9afd2d003c9"). InnerVolumeSpecName "kube-api-access-xvwl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:31:04.695242 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.695141 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-kube-api-access-c4ptg" (OuterVolumeSpecName: "kube-api-access-c4ptg") pod "1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2" (UID: "1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2"). InnerVolumeSpecName "kube-api-access-c4ptg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:31:04.794058 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.794015 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a57175f-db33-483e-8a7d-b9afd2d003c9-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:31:04.794058 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.794051 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:31:04.794058 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.794061 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4ptg\" (UniqueName: \"kubernetes.io/projected/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2-kube-api-access-c4ptg\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:31:04.794058 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:04.794069 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwl9\" (UniqueName: \"kubernetes.io/projected/2a57175f-db33-483e-8a7d-b9afd2d003c9-kube-api-access-xvwl9\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:31:05.427054 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.427019 2573 generic.go:356] "Generic (PLEG): container finished" podID="1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2" containerID="71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862" exitCode=0 Jul 27 20:31:05.427524 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.427073 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" Jul 27 20:31:05.427524 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.427107 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" event={"ID":"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2","Type":"ContainerDied","Data":"71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862"} Jul 27 20:31:05.427524 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.427164 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7" event={"ID":"1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2","Type":"ContainerDied","Data":"6523e5025e9cdb856fdc36b35b2ea74650fb7f37cad457468131d52a7f2cc267"} Jul 27 20:31:05.427524 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.427187 2573 scope.go:117] "RemoveContainer" containerID="71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862" Jul 27 20:31:05.428858 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.428840 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" Jul 27 20:31:05.428959 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.428802 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a57175f-db33-483e-8a7d-b9afd2d003c9" containerID="2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb" exitCode=0 Jul 27 20:31:05.428959 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.428869 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" event={"ID":"2a57175f-db33-483e-8a7d-b9afd2d003c9","Type":"ContainerDied","Data":"2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb"} Jul 27 20:31:05.428959 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.428914 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx" event={"ID":"2a57175f-db33-483e-8a7d-b9afd2d003c9","Type":"ContainerDied","Data":"5400fc20be37387650b9aa9a414a0d521478d112e914995180c76854fc276072"} Jul 27 20:31:05.437655 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.437637 2573 scope.go:117] "RemoveContainer" containerID="71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862" Jul 27 20:31:05.438062 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:31:05.437917 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862\": container with ID starting with 71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862 not found: ID does not exist" containerID="71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862" Jul 27 20:31:05.438062 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.437953 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862"} err="failed to get container status \"71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862\": rpc error: code = NotFound desc = could not find container \"71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862\": container with ID starting with 71e313f44a598fce61ea0a5a6e5d26824c4b4a9faa47c648818ecd2386553862 not found: ID does not exist" Jul 27 20:31:05.438062 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.437973 2573 scope.go:117] "RemoveContainer" containerID="2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb" Jul 27 20:31:05.447943 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.447922 2573 scope.go:117] "RemoveContainer" containerID="2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb" Jul 27 20:31:05.448240 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:31:05.448217 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb\": container with ID starting with 2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb not found: ID does not exist" containerID="2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb" Jul 27 20:31:05.448343 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.448248 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb"} err="failed to get container status \"2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb\": rpc error: code = NotFound desc = could not find container \"2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb\": container with ID starting with 2df13899e207178a22284a578d226af0e410fcbb0da24c5fcf1b026b4ec70eeb not found: ID does not exist" Jul 27 20:31:05.449991 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.449966 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7"] Jul 27 20:31:05.453315 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.453279 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-555352-987fc97b4-qkkw7"] Jul 27 20:31:05.463498 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.463475 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx"] Jul 27 20:31:05.465778 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.465756 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-555352-c98687c76-nlqqx"] Jul 27 20:31:05.877604 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.877574 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2" path="/var/lib/kubelet/pods/1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2/volumes" Jul 27 20:31:05.877929 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.877916 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a57175f-db33-483e-8a7d-b9afd2d003c9" path="/var/lib/kubelet/pods/2a57175f-db33-483e-8a7d-b9afd2d003c9/volumes" Jul 27 20:31:05.878152 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:05.878141 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92913cdf-ef80-4445-8282-66ceafd42704" path="/var/lib/kubelet/pods/92913cdf-ef80-4445-8282-66ceafd42704/volumes" Jul 27 20:31:08.820282 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:08.820235 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:31:08.821191 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:08.821167 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8"] Jul 27 20:31:08.823466 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:08.823446 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-555352-g5jh8"] Jul 27 20:31:09.183961 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:09.183923 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:31:09.876537 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:09.876502 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3249c41-4da0-44b5-88b7-491d1fabc275" path="/var/lib/kubelet/pods/c3249c41-4da0-44b5-88b7-491d1fabc275/volumes" Jul 27 20:31:11.931869 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:11.931834 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:31:14.270144 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:14.270091 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:31:50.312722 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:50.312686 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:31:51.631681 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:51.631642 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:31:57.530948 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:31:57.530896 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:12.540698 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:12.540657 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:12.642133 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:12.642094 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:12.819660 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:12.819546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:13.048132 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.048075 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:13.321393 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321358 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl"] Jul 27 20:32:13.321779 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321767 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2" containerName="payload-processing" Jul 27 20:32:13.321824 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321782 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2" containerName="payload-processing" Jul 27 20:32:13.321824 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321799 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="684a9029-760a-4d08-ba85-b5472b3e4d0c" containerName="registry-server" Jul 27 20:32:13.321824 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321805 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="684a9029-760a-4d08-ba85-b5472b3e4d0c" containerName="registry-server" Jul 27 20:32:13.321824 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321814 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a57175f-db33-483e-8a7d-b9afd2d003c9" containerName="payload-pre-processing" Jul 27 20:32:13.321824 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321819 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a57175f-db33-483e-8a7d-b9afd2d003c9" containerName="payload-pre-processing" Jul 27 20:32:13.321824 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321826 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3249c41-4da0-44b5-88b7-491d1fabc275" containerName="revoke-keys" Jul 27 20:32:13.322016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321831 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3249c41-4da0-44b5-88b7-491d1fabc275" containerName="revoke-keys" Jul 27 20:32:13.322016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321889 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4d9f5e-ee14-4eaf-8271-bbd3d67756b2" containerName="payload-processing" Jul 27 20:32:13.322016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321897 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="684a9029-760a-4d08-ba85-b5472b3e4d0c" containerName="registry-server" Jul 27 20:32:13.322016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321906 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3249c41-4da0-44b5-88b7-491d1fabc275" containerName="revoke-keys" Jul 27 20:32:13.322016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.321914 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a57175f-db33-483e-8a7d-b9afd2d003c9" containerName="payload-pre-processing" Jul 27 20:32:13.325026 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.325010 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" Jul 27 20:32:13.327483 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.327460 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:32:13.327662 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.327649 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:32:13.333064 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.332800 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl"] Jul 27 20:32:13.415983 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.415949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d6e65d-f936-48d8-b032-ef8ce81a1a57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl\" (UID: \"e0d6e65d-f936-48d8-b032-ef8ce81a1a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" Jul 27 20:32:13.517143 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.517103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d6e65d-f936-48d8-b032-ef8ce81a1a57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl\" (UID: \"e0d6e65d-f936-48d8-b032-ef8ce81a1a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" Jul 27 20:32:13.517778 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.517753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d6e65d-f936-48d8-b032-ef8ce81a1a57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl\" (UID: \"e0d6e65d-f936-48d8-b032-ef8ce81a1a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" Jul 27 20:32:13.637282 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.637202 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" Jul 27 20:32:13.777103 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.777079 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl"] Jul 27 20:32:13.782243 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:32:13.782175 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d6e65d_f936_48d8_b032_ef8ce81a1a57.slice/crio-78cee836917bf5b2350b1741163e2954f5fcb3d19e271b71fe5f26e5b704d452 WatchSource:0}: Error finding container 78cee836917bf5b2350b1741163e2954f5fcb3d19e271b71fe5f26e5b704d452: Status 404 returned error can't find the container with id 78cee836917bf5b2350b1741163e2954f5fcb3d19e271b71fe5f26e5b704d452 Jul 27 20:32:13.838441 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:13.838404 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" event={"ID":"e0d6e65d-f936-48d8-b032-ef8ce81a1a57","Type":"ContainerStarted","Data":"78cee836917bf5b2350b1741163e2954f5fcb3d19e271b71fe5f26e5b704d452"} Jul 27 20:32:14.845179 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:14.845073 2573 generic.go:356] "Generic (PLEG): container finished" podID="e0d6e65d-f936-48d8-b032-ef8ce81a1a57" containerID="8e3878de2a40d6cba2d643678481e8859297e8dbd87d9650240c943ef513c3eb" exitCode=0 Jul 27 20:32:14.845179 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:14.845161 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" event={"ID":"e0d6e65d-f936-48d8-b032-ef8ce81a1a57","Type":"ContainerDied","Data":"8e3878de2a40d6cba2d643678481e8859297e8dbd87d9650240c943ef513c3eb"} Jul 27 20:32:15.989451 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:15.989425 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" Jul 27 20:32:16.141665 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:16.141584 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d6e65d-f936-48d8-b032-ef8ce81a1a57-maas-api-service-ca\") pod \"e0d6e65d-f936-48d8-b032-ef8ce81a1a57\" (UID: \"e0d6e65d-f936-48d8-b032-ef8ce81a1a57\") " Jul 27 20:32:16.141947 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:16.141924 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d6e65d-f936-48d8-b032-ef8ce81a1a57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0d6e65d-f936-48d8-b032-ef8ce81a1a57" (UID: "e0d6e65d-f936-48d8-b032-ef8ce81a1a57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:32:16.242941 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:16.242905 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d6e65d-f936-48d8-b032-ef8ce81a1a57-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:16.858589 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:16.858547 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" event={"ID":"e0d6e65d-f936-48d8-b032-ef8ce81a1a57","Type":"ContainerDied","Data":"78cee836917bf5b2350b1741163e2954f5fcb3d19e271b71fe5f26e5b704d452"} Jul 27 20:32:16.858589 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:16.858598 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78cee836917bf5b2350b1741163e2954f5fcb3d19e271b71fe5f26e5b704d452" Jul 27 20:32:16.858817 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:16.858565 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl" Jul 27 20:32:24.833458 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:24.833427 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl"] Jul 27 20:32:24.838829 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:24.838804 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1517d6-29753070-rv9jl"] Jul 27 20:32:24.840620 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:24.840584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:24.860769 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:24.860739 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn"] Jul 27 20:32:24.860991 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:24.860948 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" podUID="061b627c-ca28-4a9b-8867-e6245f60aa49" containerName="payload-processing" containerID="cri-o://3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d" gracePeriod=30 Jul 27 20:32:24.869094 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:24.869068 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm"] Jul 27 20:32:24.869388 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:24.869360 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" podUID="be2489c4-2314-40ef-96f7-38d3e805eef3" containerName="payload-pre-processing" containerID="cri-o://8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda" gracePeriod=30 Jul 27 20:32:24.940189 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:24.940154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:24.998358 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:24.998323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:25.174542 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.174513 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:32:25.177223 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.177202 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:32:25.331407 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.331368 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b627c-ca28-4a9b-8867-e6245f60aa49-plugins-config-volume\") pod \"061b627c-ca28-4a9b-8867-e6245f60aa49\" (UID: \"061b627c-ca28-4a9b-8867-e6245f60aa49\") " Jul 27 20:32:25.331407 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.331413 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw59p\" (UniqueName: \"kubernetes.io/projected/061b627c-ca28-4a9b-8867-e6245f60aa49-kube-api-access-vw59p\") pod \"061b627c-ca28-4a9b-8867-e6245f60aa49\" (UID: \"061b627c-ca28-4a9b-8867-e6245f60aa49\") " Jul 27 20:32:25.331650 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.331438 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2489c4-2314-40ef-96f7-38d3e805eef3-plugins-config-volume\") pod \"be2489c4-2314-40ef-96f7-38d3e805eef3\" (UID: \"be2489c4-2314-40ef-96f7-38d3e805eef3\") " Jul 27 20:32:25.331650 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.331494 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x646k\" (UniqueName: \"kubernetes.io/projected/be2489c4-2314-40ef-96f7-38d3e805eef3-kube-api-access-x646k\") pod \"be2489c4-2314-40ef-96f7-38d3e805eef3\" (UID: \"be2489c4-2314-40ef-96f7-38d3e805eef3\") " Jul 27 20:32:25.331797 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.331770 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061b627c-ca28-4a9b-8867-e6245f60aa49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "061b627c-ca28-4a9b-8867-e6245f60aa49" (UID: "061b627c-ca28-4a9b-8867-e6245f60aa49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:32:25.331880 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.331834 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2489c4-2314-40ef-96f7-38d3e805eef3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be2489c4-2314-40ef-96f7-38d3e805eef3" (UID: "be2489c4-2314-40ef-96f7-38d3e805eef3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:32:25.333855 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.333832 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2489c4-2314-40ef-96f7-38d3e805eef3-kube-api-access-x646k" (OuterVolumeSpecName: "kube-api-access-x646k") pod "be2489c4-2314-40ef-96f7-38d3e805eef3" (UID: "be2489c4-2314-40ef-96f7-38d3e805eef3"). InnerVolumeSpecName "kube-api-access-x646k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:32:25.333855 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.333841 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061b627c-ca28-4a9b-8867-e6245f60aa49-kube-api-access-vw59p" (OuterVolumeSpecName: "kube-api-access-vw59p") pod "061b627c-ca28-4a9b-8867-e6245f60aa49" (UID: "061b627c-ca28-4a9b-8867-e6245f60aa49"). InnerVolumeSpecName "kube-api-access-vw59p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:32:25.432961 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.432930 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x646k\" (UniqueName: \"kubernetes.io/projected/be2489c4-2314-40ef-96f7-38d3e805eef3-kube-api-access-x646k\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:25.432961 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.432959 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b627c-ca28-4a9b-8867-e6245f60aa49-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:25.432961 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.432968 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw59p\" (UniqueName: \"kubernetes.io/projected/061b627c-ca28-4a9b-8867-e6245f60aa49-kube-api-access-vw59p\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:25.433200 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.432976 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be2489c4-2314-40ef-96f7-38d3e805eef3-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:25.878246 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.878161 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aefe780-6f90-42e8-93d0-1766f89c6a9b" path="/var/lib/kubelet/pods/7aefe780-6f90-42e8-93d0-1766f89c6a9b/volumes" Jul 27 20:32:25.911568 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.911537 2573 generic.go:356] "Generic (PLEG): container finished" podID="be2489c4-2314-40ef-96f7-38d3e805eef3" containerID="8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda" exitCode=0 Jul 27 20:32:25.911702 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.911580 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" Jul 27 20:32:25.911702 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.911619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" event={"ID":"be2489c4-2314-40ef-96f7-38d3e805eef3","Type":"ContainerDied","Data":"8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda"} Jul 27 20:32:25.911702 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.911665 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm" event={"ID":"be2489c4-2314-40ef-96f7-38d3e805eef3","Type":"ContainerDied","Data":"671c725bceb2ad6487b93d8c72318b8d60594dda101e905e04eb892df36dbd25"} Jul 27 20:32:25.911702 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.911688 2573 scope.go:117] "RemoveContainer" containerID="8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda" Jul 27 20:32:25.913075 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.913055 2573 generic.go:356] "Generic (PLEG): container finished" podID="061b627c-ca28-4a9b-8867-e6245f60aa49" containerID="3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d" exitCode=0 Jul 27 20:32:25.913200 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.913094 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" Jul 27 20:32:25.913200 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.913112 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" event={"ID":"061b627c-ca28-4a9b-8867-e6245f60aa49","Type":"ContainerDied","Data":"3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d"} Jul 27 20:32:25.913200 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.913159 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn" event={"ID":"061b627c-ca28-4a9b-8867-e6245f60aa49","Type":"ContainerDied","Data":"a95e0dfb3b8b6bd0533ecb623125d359336625bebee8c59d76a11904abab04e8"} Jul 27 20:32:25.921700 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.921683 2573 scope.go:117] "RemoveContainer" containerID="8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda" Jul 27 20:32:25.921973 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:32:25.921949 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda\": container with ID starting with 8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda not found: ID does not exist" containerID="8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda" Jul 27 20:32:25.922072 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.921981 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda"} err="failed to get container status \"8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda\": rpc error: code = NotFound desc = could not find container \"8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda\": container with ID starting with 8dbee2c25305795aaa3105c687ad1a9c041a179e6478477d39a4f674f701abda not found: ID does not exist" Jul 27 20:32:25.922072 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.922002 2573 scope.go:117] "RemoveContainer" containerID="3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d" Jul 27 20:32:25.932450 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.932432 2573 scope.go:117] "RemoveContainer" containerID="3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d" Jul 27 20:32:25.933377 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:32:25.933355 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d\": container with ID starting with 3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d not found: ID does not exist" containerID="3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d" Jul 27 20:32:25.933551 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.933526 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d"} err="failed to get container status \"3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d\": rpc error: code = NotFound desc = could not find container \"3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d\": container with ID starting with 3e88d13967ffe9c1fd7d20f07a5e8bb1a7ab1535f880c8eeb17b1c66a7dc227d not found: ID does not exist" Jul 27 20:32:25.933718 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.933703 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn"] Jul 27 20:32:25.936308 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.936275 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1517d6-64bffd5b6d-qg4rn"] Jul 27 20:32:25.945686 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.945666 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm"] Jul 27 20:32:25.948988 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:25.948964 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1517d6-5d4874445b-qptkm"] Jul 27 20:32:27.875681 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:27.875649 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="061b627c-ca28-4a9b-8867-e6245f60aa49" path="/var/lib/kubelet/pods/061b627c-ca28-4a9b-8867-e6245f60aa49/volumes" Jul 27 20:32:27.876140 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:27.875996 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be2489c4-2314-40ef-96f7-38d3e805eef3" path="/var/lib/kubelet/pods/be2489c4-2314-40ef-96f7-38d3e805eef3/volumes" Jul 27 20:32:28.390356 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:28.390287 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:28.392130 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:28.392105 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl"] Jul 27 20:32:28.396942 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:28.396918 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1517d6-chmtl"] Jul 27 20:32:28.767948 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:28.767910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:29.036886 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:29.036807 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:32:29.875329 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:29.875277 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d6e65d-f936-48d8-b032-ef8ce81a1a57" path="/var/lib/kubelet/pods/e0d6e65d-f936-48d8-b032-ef8ce81a1a57/volumes" Jul 27 20:32:35.088939 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.088906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:35.189401 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.189361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:35.310233 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.310197 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:35.537940 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.537891 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:35.741415 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741385 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8"] Jul 27 20:32:35.741765 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741753 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2489c4-2314-40ef-96f7-38d3e805eef3" containerName="payload-pre-processing" Jul 27 20:32:35.741816 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741766 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2489c4-2314-40ef-96f7-38d3e805eef3" containerName="payload-pre-processing" Jul 27 20:32:35.741816 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741775 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061b627c-ca28-4a9b-8867-e6245f60aa49" containerName="payload-processing" Jul 27 20:32:35.741816 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741781 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="061b627c-ca28-4a9b-8867-e6245f60aa49" containerName="payload-processing" Jul 27 20:32:35.741816 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741798 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d6e65d-f936-48d8-b032-ef8ce81a1a57" containerName="revoke-keys" Jul 27 20:32:35.741816 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741804 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d6e65d-f936-48d8-b032-ef8ce81a1a57" containerName="revoke-keys" Jul 27 20:32:35.741969 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741873 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="061b627c-ca28-4a9b-8867-e6245f60aa49" containerName="payload-processing" Jul 27 20:32:35.741969 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741880 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d6e65d-f936-48d8-b032-ef8ce81a1a57" containerName="revoke-keys" Jul 27 20:32:35.741969 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.741888 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2489c4-2314-40ef-96f7-38d3e805eef3" containerName="payload-pre-processing" Jul 27 20:32:35.744935 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.744914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" Jul 27 20:32:35.747602 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.747579 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:32:35.747736 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.747608 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\"" Jul 27 20:32:35.753741 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.753722 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8"] Jul 27 20:32:35.818433 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.818350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8\" (UID: \"d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" Jul 27 20:32:35.919016 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.918980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8\" (UID: \"d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" Jul 27 20:32:35.919647 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:35.919623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8\" (UID: \"d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" Jul 27 20:32:36.056946 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:36.056914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" Jul 27 20:32:36.194715 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:36.194690 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8"] Jul 27 20:32:36.196358 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:32:36.196332 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66b6fe1_f7de_4cd0_8a1f_0c9203ba9436.slice/crio-3861f0da4226524758491b7563dc9f9a6bf101c8e67f649641f383a6ff6ac320 WatchSource:0}: Error finding container 3861f0da4226524758491b7563dc9f9a6bf101c8e67f649641f383a6ff6ac320: Status 404 returned error can't find the container with id 3861f0da4226524758491b7563dc9f9a6bf101c8e67f649641f383a6ff6ac320 Jul 27 20:32:36.978707 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:36.978670 2573 generic.go:356] "Generic (PLEG): container finished" podID="d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436" containerID="ee8fae19a36952377552243636711e6faa990d7a5bf55ca2a8a2674f3eb07b9e" exitCode=0 Jul 27 20:32:36.978890 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:36.978759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" event={"ID":"d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436","Type":"ContainerDied","Data":"ee8fae19a36952377552243636711e6faa990d7a5bf55ca2a8a2674f3eb07b9e"} Jul 27 20:32:36.978890 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:36.978814 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" event={"ID":"d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436","Type":"ContainerStarted","Data":"3861f0da4226524758491b7563dc9f9a6bf101c8e67f649641f383a6ff6ac320"} Jul 27 20:32:38.121654 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:38.121624 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" Jul 27 20:32:38.133853 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:38.133828 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436-maas-api-service-ca\") pod \"d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436\" (UID: \"d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436\") " Jul 27 20:32:38.134211 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:38.134190 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436" (UID: "d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:32:38.235236 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:38.235140 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436-maas-api-service-ca\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:38.991467 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:38.991438 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" Jul 27 20:32:38.991649 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:38.991477 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8" event={"ID":"d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436","Type":"ContainerDied","Data":"3861f0da4226524758491b7563dc9f9a6bf101c8e67f649641f383a6ff6ac320"} Jul 27 20:32:38.991649 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:38.991524 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3861f0da4226524758491b7563dc9f9a6bf101c8e67f649641f383a6ff6ac320" Jul 27 20:32:46.289600 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.289559 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh"] Jul 27 20:32:46.294219 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.294181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:46.294418 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.294378 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c3a53e-29753070-mf9mh"] Jul 27 20:32:46.318212 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.318175 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk"] Jul 27 20:32:46.318541 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.318513 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" podUID="0670c659-f08b-4285-98a4-eccdbf780045" containerName="payload-processing" containerID="cri-o://2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b" gracePeriod=30 Jul 27 20:32:46.325596 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.325541 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj"] Jul 27 20:32:46.326068 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.326013 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" podUID="ff0ced08-2acb-4dfd-bdf7-6051ef1ce422" containerName="payload-pre-processing" containerID="cri-o://4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a" gracePeriod=30 Jul 27 20:32:46.374124 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.374090 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:46.478255 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.478164 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:46.581658 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.581630 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:32:46.606523 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.606492 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0670c659-f08b-4285-98a4-eccdbf780045-plugins-config-volume\") pod \"0670c659-f08b-4285-98a4-eccdbf780045\" (UID: \"0670c659-f08b-4285-98a4-eccdbf780045\") " Jul 27 20:32:46.606664 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.606553 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr4vp\" (UniqueName: \"kubernetes.io/projected/0670c659-f08b-4285-98a4-eccdbf780045-kube-api-access-cr4vp\") pod \"0670c659-f08b-4285-98a4-eccdbf780045\" (UID: \"0670c659-f08b-4285-98a4-eccdbf780045\") " Jul 27 20:32:46.606841 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.606817 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0670c659-f08b-4285-98a4-eccdbf780045-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0670c659-f08b-4285-98a4-eccdbf780045" (UID: "0670c659-f08b-4285-98a4-eccdbf780045"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:32:46.608897 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.608870 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0670c659-f08b-4285-98a4-eccdbf780045-kube-api-access-cr4vp" (OuterVolumeSpecName: "kube-api-access-cr4vp") pod "0670c659-f08b-4285-98a4-eccdbf780045" (UID: "0670c659-f08b-4285-98a4-eccdbf780045"). InnerVolumeSpecName "kube-api-access-cr4vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:32:46.610983 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.610963 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:32:46.707892 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.707856 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2kb8\" (UniqueName: \"kubernetes.io/projected/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-kube-api-access-n2kb8\") pod \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\" (UID: \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\") " Jul 27 20:32:46.708096 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.707948 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-plugins-config-volume\") pod \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\" (UID: \"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422\") " Jul 27 20:32:46.708214 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.708196 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0670c659-f08b-4285-98a4-eccdbf780045-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:46.708271 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.708221 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr4vp\" (UniqueName: \"kubernetes.io/projected/0670c659-f08b-4285-98a4-eccdbf780045-kube-api-access-cr4vp\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:46.708357 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.708286 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff0ced08-2acb-4dfd-bdf7-6051ef1ce422" (UID: "ff0ced08-2acb-4dfd-bdf7-6051ef1ce422"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:32:46.710136 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.710115 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-kube-api-access-n2kb8" (OuterVolumeSpecName: "kube-api-access-n2kb8") pod "ff0ced08-2acb-4dfd-bdf7-6051ef1ce422" (UID: "ff0ced08-2acb-4dfd-bdf7-6051ef1ce422"). InnerVolumeSpecName "kube-api-access-n2kb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:32:46.809422 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.809388 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2kb8\" (UniqueName: \"kubernetes.io/projected/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-kube-api-access-n2kb8\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:46.809422 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:46.809413 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:32:47.038984 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.038943 2573 generic.go:356] "Generic (PLEG): container finished" podID="ff0ced08-2acb-4dfd-bdf7-6051ef1ce422" containerID="4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a" exitCode=0 Jul 27 20:32:47.039174 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.038995 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" Jul 27 20:32:47.039174 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.039028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" event={"ID":"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422","Type":"ContainerDied","Data":"4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a"} Jul 27 20:32:47.039174 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.039083 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj" event={"ID":"ff0ced08-2acb-4dfd-bdf7-6051ef1ce422","Type":"ContainerDied","Data":"a22d8c2d4f007f4d9dd11af95707f04cfd4e122d8e5aab67320a26b93b4cd305"} Jul 27 20:32:47.039174 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.039115 2573 scope.go:117] "RemoveContainer" containerID="4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a" Jul 27 20:32:47.040549 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.040525 2573 generic.go:356] "Generic (PLEG): container finished" podID="0670c659-f08b-4285-98a4-eccdbf780045" containerID="2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b" exitCode=0 Jul 27 20:32:47.040662 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.040604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" event={"ID":"0670c659-f08b-4285-98a4-eccdbf780045","Type":"ContainerDied","Data":"2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b"} Jul 27 20:32:47.040662 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.040654 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" event={"ID":"0670c659-f08b-4285-98a4-eccdbf780045","Type":"ContainerDied","Data":"97ee09300e4969bb96d13a2130d54575b289c68202f809e9cf6c43c530026711"} Jul 27 20:32:47.040746 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.040614 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk" Jul 27 20:32:47.050334 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.050314 2573 scope.go:117] "RemoveContainer" containerID="4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a" Jul 27 20:32:47.050697 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:32:47.050672 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a\": container with ID starting with 4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a not found: ID does not exist" containerID="4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a" Jul 27 20:32:47.050800 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.050711 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a"} err="failed to get container status \"4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a\": rpc error: code = NotFound desc = could not find container \"4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a\": container with ID starting with 4d00e5d94de8f4020ac3546f4424f8d4cce2a87ca4267084a297de6be1a3431a not found: ID does not exist" Jul 27 20:32:47.050800 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.050734 2573 scope.go:117] "RemoveContainer" containerID="2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b" Jul 27 20:32:47.059942 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.059915 2573 scope.go:117] "RemoveContainer" containerID="2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b" Jul 27 20:32:47.060229 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:32:47.060209 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b\": container with ID starting with 2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b not found: ID does not exist" containerID="2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b" Jul 27 20:32:47.060289 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.060239 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b"} err="failed to get container status \"2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b\": rpc error: code = NotFound desc = could not find container \"2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b\": container with ID starting with 2811d454b30509461e31f183c2d411fc995b93d70b353f42da2b1bf0f332842b not found: ID does not exist" Jul 27 20:32:47.062060 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.062040 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj"] Jul 27 20:32:47.065763 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.065744 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c3a53e-746b6644c6-srrnj"] Jul 27 20:32:47.075879 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.075859 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk"] Jul 27 20:32:47.079390 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.079370 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c3a53e-56b5cb77d-pdshk"] Jul 27 20:32:47.875211 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.875174 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0670c659-f08b-4285-98a4-eccdbf780045" path="/var/lib/kubelet/pods/0670c659-f08b-4285-98a4-eccdbf780045/volumes" Jul 27 20:32:47.875613 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.875477 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75323609-4b13-424d-ab83-5585ed6f084d" path="/var/lib/kubelet/pods/75323609-4b13-424d-ab83-5585ed6f084d/volumes" Jul 27 20:32:47.875754 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:47.875740 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff0ced08-2acb-4dfd-bdf7-6051ef1ce422" path="/var/lib/kubelet/pods/ff0ced08-2acb-4dfd-bdf7-6051ef1ce422/volumes" Jul 27 20:32:49.731958 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:49.731917 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:32:50.972339 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:50.972291 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8"] Jul 27 20:32:50.976392 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:50.976356 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:50.977251 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:50.977232 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c3a53e-6nst8"] Jul 27 20:32:51.331142 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:51.331041 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:51.876700 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:51.876664 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436" path="/var/lib/kubelet/pods/d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436/volumes" Jul 27 20:32:57.236531 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:57.236495 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:57.336277 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:57.336241 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:57.479907 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:57.479871 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:32:57.709024 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:32:57.708992 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:09.292024 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.291984 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn"] Jul 27 20:33:09.294691 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.294657 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f44149-29753070-6znwn"] Jul 27 20:33:09.295082 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.295061 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:09.320034 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.319997 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k"] Jul 27 20:33:09.320260 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.320217 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" podUID="9ce38ce0-149f-446f-adc0-8088696ccac8" containerName="payload-processing" containerID="cri-o://b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c" gracePeriod=30 Jul 27 20:33:09.331917 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.331873 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg"] Jul 27 20:33:09.332144 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.332117 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" podUID="8d5d2911-9880-4497-a887-d7c2b53a2bfc" containerName="payload-pre-processing" containerID="cri-o://7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9" gracePeriod=30 Jul 27 20:33:09.363711 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.363673 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:09.427471 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.427422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:09.615264 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.615234 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:33:09.618621 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.618595 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:33:09.719738 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.719693 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d2911-9880-4497-a887-d7c2b53a2bfc-plugins-config-volume\") pod \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\" (UID: \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\") " Jul 27 20:33:09.719738 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.719737 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljvkz\" (UniqueName: \"kubernetes.io/projected/9ce38ce0-149f-446f-adc0-8088696ccac8-kube-api-access-ljvkz\") pod \"9ce38ce0-149f-446f-adc0-8088696ccac8\" (UID: \"9ce38ce0-149f-446f-adc0-8088696ccac8\") " Jul 27 20:33:09.720014 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.719799 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce38ce0-149f-446f-adc0-8088696ccac8-plugins-config-volume\") pod \"9ce38ce0-149f-446f-adc0-8088696ccac8\" (UID: \"9ce38ce0-149f-446f-adc0-8088696ccac8\") " Jul 27 20:33:09.720014 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.719848 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2rkl\" (UniqueName: \"kubernetes.io/projected/8d5d2911-9880-4497-a887-d7c2b53a2bfc-kube-api-access-g2rkl\") pod \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\" (UID: \"8d5d2911-9880-4497-a887-d7c2b53a2bfc\") " Jul 27 20:33:09.720172 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.720143 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5d2911-9880-4497-a887-d7c2b53a2bfc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d5d2911-9880-4497-a887-d7c2b53a2bfc" (UID: "8d5d2911-9880-4497-a887-d7c2b53a2bfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:33:09.720232 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.720180 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce38ce0-149f-446f-adc0-8088696ccac8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ce38ce0-149f-446f-adc0-8088696ccac8" (UID: "9ce38ce0-149f-446f-adc0-8088696ccac8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:33:09.722044 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.722023 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce38ce0-149f-446f-adc0-8088696ccac8-kube-api-access-ljvkz" (OuterVolumeSpecName: "kube-api-access-ljvkz") pod "9ce38ce0-149f-446f-adc0-8088696ccac8" (UID: "9ce38ce0-149f-446f-adc0-8088696ccac8"). InnerVolumeSpecName "kube-api-access-ljvkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:33:09.722044 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.722029 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5d2911-9880-4497-a887-d7c2b53a2bfc-kube-api-access-g2rkl" (OuterVolumeSpecName: "kube-api-access-g2rkl") pod "8d5d2911-9880-4497-a887-d7c2b53a2bfc" (UID: "8d5d2911-9880-4497-a887-d7c2b53a2bfc"). InnerVolumeSpecName "kube-api-access-g2rkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:33:09.820831 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.820781 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d2911-9880-4497-a887-d7c2b53a2bfc-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:33:09.820831 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.820820 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljvkz\" (UniqueName: \"kubernetes.io/projected/9ce38ce0-149f-446f-adc0-8088696ccac8-kube-api-access-ljvkz\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:33:09.820831 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.820834 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce38ce0-149f-446f-adc0-8088696ccac8-plugins-config-volume\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:33:09.821079 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.820847 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rkl\" (UniqueName: \"kubernetes.io/projected/8d5d2911-9880-4497-a887-d7c2b53a2bfc-kube-api-access-g2rkl\") on node \"ip-10-0-141-6.ec2.internal\" DevicePath \"\"" Jul 27 20:33:09.875423 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:09.875336 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872cc807-7937-4545-b19f-58ce352284e1" path="/var/lib/kubelet/pods/872cc807-7937-4545-b19f-58ce352284e1/volumes" Jul 27 20:33:10.177776 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.177678 2573 generic.go:356] "Generic (PLEG): container finished" podID="8d5d2911-9880-4497-a887-d7c2b53a2bfc" containerID="7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9" exitCode=0 Jul 27 20:33:10.177776 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.177723 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" Jul 27 20:33:10.177776 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.177761 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" event={"ID":"8d5d2911-9880-4497-a887-d7c2b53a2bfc","Type":"ContainerDied","Data":"7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9"} Jul 27 20:33:10.178057 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.177819 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg" event={"ID":"8d5d2911-9880-4497-a887-d7c2b53a2bfc","Type":"ContainerDied","Data":"986324f98d72411ae675a8a12f1a4ee400354d19c62fbda078ae2c7011ff005b"} Jul 27 20:33:10.178057 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.177836 2573 scope.go:117] "RemoveContainer" containerID="7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9" Jul 27 20:33:10.179227 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.179202 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ce38ce0-149f-446f-adc0-8088696ccac8" containerID="b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c" exitCode=0 Jul 27 20:33:10.179336 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.179243 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" Jul 27 20:33:10.179336 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.179290 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" event={"ID":"9ce38ce0-149f-446f-adc0-8088696ccac8","Type":"ContainerDied","Data":"b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c"} Jul 27 20:33:10.179434 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.179374 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k" event={"ID":"9ce38ce0-149f-446f-adc0-8088696ccac8","Type":"ContainerDied","Data":"9441b94bc1afaf637f99cf472d2e800ac8909134c5045df4f72689ec5653dd66"} Jul 27 20:33:10.187559 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.187518 2573 scope.go:117] "RemoveContainer" containerID="7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9" Jul 27 20:33:10.187894 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:33:10.187873 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9\": container with ID starting with 7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9 not found: ID does not exist" containerID="7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9" Jul 27 20:33:10.187948 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.187904 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9"} err="failed to get container status \"7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9\": rpc error: code = NotFound desc = could not find container \"7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9\": container with ID starting with 7fb85735aa7cca0c29c697490196d33c8ffa4359451c24d2783f35af9290edb9 not found: ID does not exist" Jul 27 20:33:10.187948 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.187922 2573 scope.go:117] "RemoveContainer" containerID="b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c" Jul 27 20:33:10.197927 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.197904 2573 scope.go:117] "RemoveContainer" containerID="b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c" Jul 27 20:33:10.198191 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.198169 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg"] Jul 27 20:33:10.198340 ip-10-0-141-6 kubenswrapper[2573]: E0727 20:33:10.198254 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c\": container with ID starting with b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c not found: ID does not exist" containerID="b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c" Jul 27 20:33:10.198340 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.198282 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c"} err="failed to get container status \"b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c\": rpc error: code = NotFound desc = could not find container \"b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c\": container with ID starting with b071471061404c786c3360969983709fd48b49343455c541c17b13439232845c not found: ID does not exist" Jul 27 20:33:10.200364 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.200342 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f44149-5dd56d7784-jqdmg"] Jul 27 20:33:10.208672 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.208639 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k"] Jul 27 20:33:10.212002 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:10.211972 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f44149-54bdcf7dcd-dd59k"] Jul 27 20:33:11.875446 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:11.875409 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5d2911-9880-4497-a887-d7c2b53a2bfc" path="/var/lib/kubelet/pods/8d5d2911-9880-4497-a887-d7c2b53a2bfc/volumes" Jul 27 20:33:11.875829 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:11.875754 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ce38ce0-149f-446f-adc0-8088696ccac8" path="/var/lib/kubelet/pods/9ce38ce0-149f-446f-adc0-8088696ccac8/volumes" Jul 27 20:33:12.128943 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:12.128859 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fj8gw"] Jul 27 20:33:12.979503 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:12.979460 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:13.361787 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:13.361705 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:19.262569 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:19.262531 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:19.352847 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:19.352807 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:19.490100 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:19.490059 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:19.707380 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:19.707341 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:24.931141 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:24.931105 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:24.987135 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:24.987100 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:26.227498 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:26.227440 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df5588f67-zxvlz_02970dfb-7f1c-4344-814c-8efe0ff90515/copy-manifests/0.log" Jul 27 20:33:26.235238 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:26.235192 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df5588f67-zxvlz_02970dfb-7f1c-4344-814c-8efe0ff90515/manager/0.log" Jul 27 20:33:26.819689 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:26.819654 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753055-hjnc8_ac9298fa-ca59-4153-a89b-f794b5da8b0e/cleanup/0.log" Jul 27 20:33:27.783215 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:27.783177 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm_7e4e3756-4ce4-4d91-b3b2-4d44008779bd/util/0.log" Jul 27 20:33:27.790813 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:27.790788 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm_7e4e3756-4ce4-4d91-b3b2-4d44008779bd/pull/0.log" Jul 27 20:33:27.797033 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:27.796995 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm_7e4e3756-4ce4-4d91-b3b2-4d44008779bd/extract/0.log" Jul 27 20:33:27.901864 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:27.901837 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw_49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9/util/0.log" Jul 27 20:33:27.907698 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:27.907662 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw_49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9/pull/0.log" Jul 27 20:33:27.913089 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:27.913063 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw_49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9/extract/0.log" Jul 27 20:33:28.019279 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:28.019254 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s_d341abe1-ac9b-4c10-8171-9dcdf9be1ecf/util/0.log" Jul 27 20:33:28.025938 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:28.025894 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s_d341abe1-ac9b-4c10-8171-9dcdf9be1ecf/pull/0.log" Jul 27 20:33:28.031870 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:28.031839 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s_d341abe1-ac9b-4c10-8171-9dcdf9be1ecf/extract/0.log" Jul 27 20:33:28.134087 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:28.133985 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh_13ec8e0e-df8f-418e-9c91-691c7f22082e/pull/0.log" Jul 27 20:33:28.139444 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:28.139419 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh_13ec8e0e-df8f-418e-9c91-691c7f22082e/extract/0.log" Jul 27 20:33:28.144930 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:28.144904 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh_13ec8e0e-df8f-418e-9c91-691c7f22082e/util/0.log" Jul 27 20:33:28.923767 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:28.923737 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fj8gw_c0a79376-660e-45b2-a744-9703fc2cc8df/limitador/0.log" Jul 27 20:33:29.510394 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:29.510361 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-snmmk_642a4ebe-1b8e-4003-984e-74a01f4827d8/discovery/0.log" Jul 27 20:33:30.715987 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:30.715951 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-8nqfm_d078e159-4ecb-4a0f-b645-5ec21c4225c0/main/0.log" Jul 27 20:33:30.932108 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:30.932074 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hrqhm_1b0f13f4-0fa8-4973-999b-c7e19dbbc95c/main/0.log" Jul 27 20:33:36.356853 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:36.356812 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:37.509744 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:37.509712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" limit=1024 Jul 27 20:33:39.215831 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:39.215769 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-69k4m_f9911a42-c569-4d6b-89e9-08a1551fd7cc/global-pull-secret-syncer/0.log" Jul 27 20:33:39.360792 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:39.360756 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fs6t5_4cfed4dd-9be6-4717-889d-7813b1dceb33/konnectivity-agent/0.log" Jul 27 20:33:39.490442 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:39.490320 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-6.ec2.internal_a05271190b4f7ec0b7de75b83b48e465/haproxy/0.log" Jul 27 20:33:43.524801 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.524767 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm_7e4e3756-4ce4-4d91-b3b2-4d44008779bd/extract/0.log" Jul 27 20:33:43.546869 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.546836 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm_7e4e3756-4ce4-4d91-b3b2-4d44008779bd/util/0.log" Jul 27 20:33:43.570749 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.570718 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hpfmm_7e4e3756-4ce4-4d91-b3b2-4d44008779bd/pull/0.log" Jul 27 20:33:43.643647 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.643614 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw_49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9/extract/0.log" Jul 27 20:33:43.687802 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.687767 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw_49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9/util/0.log" Jul 27 20:33:43.752057 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.752021 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tvsmw_49b7b3cc-97c2-4ebc-8b94-a462a3daf3b9/pull/0.log" Jul 27 20:33:43.790739 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.790654 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s_d341abe1-ac9b-4c10-8171-9dcdf9be1ecf/extract/0.log" Jul 27 20:33:43.811243 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.811215 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s_d341abe1-ac9b-4c10-8171-9dcdf9be1ecf/util/0.log" Jul 27 20:33:43.832067 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.832036 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cbj8s_d341abe1-ac9b-4c10-8171-9dcdf9be1ecf/pull/0.log" Jul 27 20:33:43.858785 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.858755 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh_13ec8e0e-df8f-418e-9c91-691c7f22082e/extract/0.log" Jul 27 20:33:43.880887 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.880859 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh_13ec8e0e-df8f-418e-9c91-691c7f22082e/util/0.log" Jul 27 20:33:43.904503 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:43.904474 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef175pwh_13ec8e0e-df8f-418e-9c91-691c7f22082e/pull/0.log" Jul 27 20:33:44.939569 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:44.939538 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fj8gw_c0a79376-660e-45b2-a744-9703fc2cc8df/limitador/0.log" Jul 27 20:33:46.785619 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:46.785575 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f849de3-35a1-40b9-a7b1-817e1ea3a0f0/alertmanager/0.log" Jul 27 20:33:46.823739 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:46.823709 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f849de3-35a1-40b9-a7b1-817e1ea3a0f0/config-reloader/0.log" Jul 27 20:33:46.852532 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:46.852488 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f849de3-35a1-40b9-a7b1-817e1ea3a0f0/kube-rbac-proxy-web/0.log" Jul 27 20:33:46.882278 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:46.882254 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f849de3-35a1-40b9-a7b1-817e1ea3a0f0/kube-rbac-proxy/0.log" Jul 27 20:33:46.905601 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:46.905525 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f849de3-35a1-40b9-a7b1-817e1ea3a0f0/kube-rbac-proxy-metric/0.log" Jul 27 20:33:46.943390 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:46.943361 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f849de3-35a1-40b9-a7b1-817e1ea3a0f0/prom-label-proxy/0.log" Jul 27 20:33:46.970089 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:46.970054 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f849de3-35a1-40b9-a7b1-817e1ea3a0f0/init-config-reloader/0.log" Jul 27 20:33:47.053532 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.053497 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zr86l_7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8/kube-state-metrics/0.log" Jul 27 20:33:47.077357 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.077324 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zr86l_7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8/kube-rbac-proxy-main/0.log" Jul 27 20:33:47.101776 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.101746 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zr86l_7d9d2e21-5dff-4ad8-bf15-e3bd4f44e2b8/kube-rbac-proxy-self/0.log" Jul 27 20:33:47.131727 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.131702 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-566b9db6d4-pxqr5_be0f1e17-ddc8-4d05-895b-4ca6a9504f76/metrics-server/0.log" Jul 27 20:33:47.281755 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.281719 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rk58b_37dda389-7aa6-461e-8fcb-9d00ae6dde05/node-exporter/0.log" Jul 27 20:33:47.307687 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.307659 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rk58b_37dda389-7aa6-461e-8fcb-9d00ae6dde05/kube-rbac-proxy/0.log" Jul 27 20:33:47.333384 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.333356 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rk58b_37dda389-7aa6-461e-8fcb-9d00ae6dde05/init-textfile/0.log" Jul 27 20:33:47.451195 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.451153 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-tx756_88007507-fc7d-4645-973d-2eb422dabc19/kube-rbac-proxy-main/0.log" Jul 27 20:33:47.476811 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.476782 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-tx756_88007507-fc7d-4645-973d-2eb422dabc19/kube-rbac-proxy-self/0.log" Jul 27 20:33:47.504496 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.504453 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-tx756_88007507-fc7d-4645-973d-2eb422dabc19/openshift-state-metrics/0.log" Jul 27 20:33:47.730685 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.730653 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-hk4nc_b9d57a89-726b-4797-a5b4-b20af5d66859/prometheus-operator/0.log" Jul 27 20:33:47.752540 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.752512 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-hk4nc_b9d57a89-726b-4797-a5b4-b20af5d66859/kube-rbac-proxy/0.log" Jul 27 20:33:47.779138 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.779114 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-cz6gw_398a7dd4-7a0a-4c74-9402-dd6c6c7857ac/prometheus-operator-admission-webhook/0.log" Jul 27 20:33:47.879594 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.879562 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d975c7cc9-gx82d_cb4a5e90-1688-4ab4-b788-1a14309cd7b9/thanos-query/0.log" Jul 27 20:33:47.908554 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.908525 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d975c7cc9-gx82d_cb4a5e90-1688-4ab4-b788-1a14309cd7b9/kube-rbac-proxy-web/0.log" Jul 27 20:33:47.929376 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.929345 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d975c7cc9-gx82d_cb4a5e90-1688-4ab4-b788-1a14309cd7b9/kube-rbac-proxy/0.log" Jul 27 20:33:47.951826 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.951793 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d975c7cc9-gx82d_cb4a5e90-1688-4ab4-b788-1a14309cd7b9/prom-label-proxy/0.log" Jul 27 20:33:47.972578 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.972539 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d975c7cc9-gx82d_cb4a5e90-1688-4ab4-b788-1a14309cd7b9/kube-rbac-proxy-rules/0.log" Jul 27 20:33:47.993997 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:47.993914 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d975c7cc9-gx82d_cb4a5e90-1688-4ab4-b788-1a14309cd7b9/kube-rbac-proxy-metrics/0.log" Jul 27 20:33:49.282549 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.282519 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp"] Jul 27 20:33:49.283052 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283032 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce38ce0-149f-446f-adc0-8088696ccac8" containerName="payload-processing" Jul 27 20:33:49.283135 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283054 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce38ce0-149f-446f-adc0-8088696ccac8" containerName="payload-processing" Jul 27 20:33:49.283135 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283073 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff0ced08-2acb-4dfd-bdf7-6051ef1ce422" containerName="payload-pre-processing" Jul 27 20:33:49.283135 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283081 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff0ced08-2acb-4dfd-bdf7-6051ef1ce422" containerName="payload-pre-processing" Jul 27 20:33:49.283135 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283103 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5d2911-9880-4497-a887-d7c2b53a2bfc" containerName="payload-pre-processing" Jul 27 20:33:49.283135 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283111 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5d2911-9880-4497-a887-d7c2b53a2bfc" containerName="payload-pre-processing" Jul 27 20:33:49.283135 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283128 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436" containerName="revoke-keys" Jul 27 20:33:49.283135 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283137 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436" containerName="revoke-keys" Jul 27 20:33:49.283504 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283150 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0670c659-f08b-4285-98a4-eccdbf780045" containerName="payload-processing" Jul 27 20:33:49.283504 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283159 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0670c659-f08b-4285-98a4-eccdbf780045" containerName="payload-processing" Jul 27 20:33:49.283504 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283248 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66b6fe1-f7de-4cd0-8a1f-0c9203ba9436" containerName="revoke-keys" Jul 27 20:33:49.283504 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283263 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5d2911-9880-4497-a887-d7c2b53a2bfc" containerName="payload-pre-processing" Jul 27 20:33:49.283504 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283276 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0670c659-f08b-4285-98a4-eccdbf780045" containerName="payload-processing" Jul 27 20:33:49.283504 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283288 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff0ced08-2acb-4dfd-bdf7-6051ef1ce422" containerName="payload-pre-processing" Jul 27 20:33:49.283504 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.283318 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce38ce0-149f-446f-adc0-8088696ccac8" containerName="payload-processing" Jul 27 20:33:49.286556 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.286535 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.288994 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.288974 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sgjdf\"/\"openshift-service-ca.crt\"" Jul 27 20:33:49.289353 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.289331 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sgjdf\"/\"kube-root-ca.crt\"" Jul 27 20:33:49.290288 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.290270 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sgjdf\"/\"default-dockercfg-spjbp\"" Jul 27 20:33:49.301422 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.301358 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp"] Jul 27 20:33:49.359894 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.359858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-proc\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.360085 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.359897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-lib-modules\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.360085 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.360017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-podres\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.360085 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.360050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-sys\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.360085 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.360067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grv7k\" (UniqueName: \"kubernetes.io/projected/cc943924-bc0d-48e6-aba6-6f81e85c02fa-kube-api-access-grv7k\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.461265 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.461234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-podres\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.461469 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.461277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-sys\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.461469 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.461324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grv7k\" (UniqueName: \"kubernetes.io/projected/cc943924-bc0d-48e6-aba6-6f81e85c02fa-kube-api-access-grv7k\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.461469 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.461371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-proc\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.461469 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.461395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-lib-modules\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.461469 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.461431 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-podres\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.461469 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.461460 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-proc\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.461682 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.461396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-sys\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.461682 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.461538 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc943924-bc0d-48e6-aba6-6f81e85c02fa-lib-modules\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.469963 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.469928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grv7k\" (UniqueName: \"kubernetes.io/projected/cc943924-bc0d-48e6-aba6-6f81e85c02fa-kube-api-access-grv7k\") pod \"perf-node-gather-daemonset-xxmpp\" (UID: \"cc943924-bc0d-48e6-aba6-6f81e85c02fa\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.599897 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.599794 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:49.738259 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:49.738173 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp"] Jul 27 20:33:49.746572 ip-10-0-141-6 kubenswrapper[2573]: W0727 20:33:49.746537 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcc943924_bc0d_48e6_aba6_6f81e85c02fa.slice/crio-452d0b6eba787573f06c73d2bcfef75e2fa879fdcfd6f1475ef6f2a413ddd856 WatchSource:0}: Error finding container 452d0b6eba787573f06c73d2bcfef75e2fa879fdcfd6f1475ef6f2a413ddd856: Status 404 returned error can't find the container with id 452d0b6eba787573f06c73d2bcfef75e2fa879fdcfd6f1475ef6f2a413ddd856 Jul 27 20:33:50.411049 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:50.411013 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" event={"ID":"cc943924-bc0d-48e6-aba6-6f81e85c02fa","Type":"ContainerStarted","Data":"188ccb378e8728bb878034a33259000620b5e226a5fba3cdc1c8a6cca15c140c"} Jul 27 20:33:50.411567 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:50.411070 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" event={"ID":"cc943924-bc0d-48e6-aba6-6f81e85c02fa","Type":"ContainerStarted","Data":"452d0b6eba787573f06c73d2bcfef75e2fa879fdcfd6f1475ef6f2a413ddd856"} Jul 27 20:33:50.411567 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:50.411145 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:50.428168 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:50.428116 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" podStartSLOduration=1.4281015400000001 podStartE2EDuration="1.42810154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:33:50.426563717 +0000 UTC m=+3235.039148303" watchObservedRunningTime="2026-07-27 20:33:50.42810154 +0000 UTC m=+3235.040686123" Jul 27 20:33:52.075040 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:52.075013 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crfhl_093af648-a209-43a7-bfb4-16b0155ae839/dns/0.log" Jul 27 20:33:52.096026 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:52.096001 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crfhl_093af648-a209-43a7-bfb4-16b0155ae839/kube-rbac-proxy/0.log" Jul 27 20:33:52.162662 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:52.162636 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-55zm9_087c69da-4dc1-4b01-9b0b-f972323a4124/dns-node-resolver/0.log" Jul 27 20:33:52.786839 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:52.786799 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wmbgb_5cf1c32f-1dc1-45e1-8992-6443f5ea0df1/node-ca/0.log" Jul 27 20:33:53.971967 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:53.971935 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-snmmk_642a4ebe-1b8e-4003-984e-74a01f4827d8/discovery/0.log" Jul 27 20:33:54.955427 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:54.955386 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5gvl7_c17ba55d-0199-45e1-b144-837dd9760c6e/serve-healthcheck-canary/0.log" Jul 27 20:33:55.621091 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:55.621053 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tzhgg_baa95d10-67aa-49d1-8e70-a7d0e2652e95/kube-rbac-proxy/0.log" Jul 27 20:33:55.640829 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:55.640797 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tzhgg_baa95d10-67aa-49d1-8e70-a7d0e2652e95/exporter/0.log" Jul 27 20:33:55.663676 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:55.663638 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tzhgg_baa95d10-67aa-49d1-8e70-a7d0e2652e95/extractor/0.log" Jul 27 20:33:56.427770 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:56.427737 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-xxmpp" Jul 27 20:33:58.743480 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:58.743386 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df5588f67-zxvlz_02970dfb-7f1c-4344-814c-8efe0ff90515/manager/0.log" Jul 27 20:33:58.764718 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:33:58.764693 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df5588f67-zxvlz_02970dfb-7f1c-4344-814c-8efe0ff90515/copy-manifests/0.log" Jul 27 20:34:00.081485 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:00.081447 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8689f9464-r7dpm_e0ab9eff-a9f1-444a-bec3-0efe9c991f9a/manager/0.log" Jul 27 20:34:00.107726 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:00.107689 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-h8bkb_9c3436a5-575a-424b-8c7a-2c1418e8ea27/openshift-lws-operator/0.log" Jul 27 20:34:00.353599 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:00.353570 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-snmmk_642a4ebe-1b8e-4003-984e-74a01f4827d8/discovery/0.log" Jul 27 20:34:02.168849 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:02.168807 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-snmmk_642a4ebe-1b8e-4003-984e-74a01f4827d8/discovery/0.log" Jul 27 20:34:06.626971 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:06.626943 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wm7nj_1873aafc-ad68-4fc5-9c04-f6070959451b/kube-multus-additional-cni-plugins/0.log" Jul 27 20:34:06.663832 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:06.663804 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wm7nj_1873aafc-ad68-4fc5-9c04-f6070959451b/egress-router-binary-copy/0.log" Jul 27 20:34:06.704880 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:06.704850 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wm7nj_1873aafc-ad68-4fc5-9c04-f6070959451b/cni-plugins/0.log" Jul 27 20:34:06.744982 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:06.744952 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wm7nj_1873aafc-ad68-4fc5-9c04-f6070959451b/bond-cni-plugin/0.log" Jul 27 20:34:06.784694 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:06.784662 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wm7nj_1873aafc-ad68-4fc5-9c04-f6070959451b/routeoverride-cni/0.log" Jul 27 20:34:06.828888 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:06.828861 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wm7nj_1873aafc-ad68-4fc5-9c04-f6070959451b/whereabouts-cni-bincopy/0.log" Jul 27 20:34:06.863809 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:06.863775 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wm7nj_1873aafc-ad68-4fc5-9c04-f6070959451b/whereabouts-cni/0.log" Jul 27 20:34:06.941336 ip-10-0-141-6 kubenswrapper[2573]: I0727 20:34:06.941279 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bdx6w_312ad691-dc6a-4105-9390-b32089a8ca6d/kube-multus/0.log"