Jul 29 14:01:14.216730 ip-10-0-130-28 systemd[1]: Starting Kubernetes Kubelet... Jul 29 14:01:14.629880 ip-10-0-130-28 kubenswrapper[2549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 14:01:14.629880 ip-10-0-130-28 kubenswrapper[2549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 14:01:14.629880 ip-10-0-130-28 kubenswrapper[2549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 14:01:14.629880 ip-10-0-130-28 kubenswrapper[2549]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 14:01:14.629880 ip-10-0-130-28 kubenswrapper[2549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 14:01:14.632321 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.632208 2549 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 14:01:14.635963 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635940 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:14.635963 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635962 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:14.635963 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635966 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635969 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635974 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635977 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635980 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635983 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635986 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635989 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635992 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635994 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.635997 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636000 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636003 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636006 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636008 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636011 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636014 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636016 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636020 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636023 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:14.636065 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636025 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636028 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636031 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636033 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636036 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636038 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636041 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636044 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636049 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636054 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636057 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636060 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636063 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636066 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636069 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636072 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636075 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636078 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636080 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:14.636560 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636083 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636086 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636088 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636091 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636094 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636096 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636099 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636102 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636105 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636107 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636109 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636112 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636114 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636117 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636120 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636123 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636126 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636128 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636131 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:14.637026 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636135 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636139 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636142 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636145 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636149 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636152 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636156 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636159 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636161 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636164 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636167 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636169 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636172 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636174 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636177 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636179 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636182 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636184 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636195 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636197 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:14.637505 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636200 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636203 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636205 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636208 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636210 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636213 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636216 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636218 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636221 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636224 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636227 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636229 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636232 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636234 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636237 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636239 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636242 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636244 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636248 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.636998 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:14.637988 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637004 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637008 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637011 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637013 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637016 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637018 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637021 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637024 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637027 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637031 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637035 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637040 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637045 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637048 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637051 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637054 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637057 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637060 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637062 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:14.638511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637065 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637069 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637072 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637075 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637078 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637081 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637084 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637087 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637090 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637093 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637096 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637099 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637102 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637105 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637108 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637111 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637114 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637117 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637120 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:14.638996 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637122 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637125 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637127 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637130 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637132 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637135 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637137 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637140 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637143 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637145 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637148 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637151 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637153 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637156 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637159 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637161 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637164 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637167 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637169 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637171 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:14.639498 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637174 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637177 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637180 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637183 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637186 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637189 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637191 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637194 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637196 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637199 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637202 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637204 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637207 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637209 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637212 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637214 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637217 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637219 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637222 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637225 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:14.640000 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637227 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637232 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637235 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637238 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637241 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637243 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637246 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637248 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637253 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637256 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637258 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637261 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637264 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637267 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637270 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637272 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637275 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637277 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637279 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637282 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:14.640506 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637699 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637705 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637708 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637711 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637713 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637716 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637719 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637721 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637724 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637727 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637730 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637733 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637736 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637739 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637743 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637745 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637748 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637751 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637753 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:14.641138 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637756 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637758 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637761 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637764 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637766 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637769 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637771 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637774 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637776 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637779 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637782 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637784 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637787 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637789 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637792 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637794 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637797 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637799 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637801 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637804 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:14.641627 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637807 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637809 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637812 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637814 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637817 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637820 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637824 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637829 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637831 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637834 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637837 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637839 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637843 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637845 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637848 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637850 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637853 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637855 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637858 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:14.642121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637860 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637863 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637865 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637868 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637871 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637873 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637876 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637878 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637881 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637883 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637885 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637888 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637890 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637894 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637898 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637901 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637904 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637908 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637910 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637913 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:14.642645 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637916 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637919 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637921 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637924 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637926 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637929 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637933 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637936 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637939 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637942 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637944 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637947 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637949 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637952 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637954 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637957 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637960 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637962 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637965 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637967 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:14.643161 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.637970 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638053 2549 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638062 2549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638068 2549 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638073 2549 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638078 2549 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638081 2549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638085 2549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638091 2549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638094 2549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638097 2549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638101 2549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638106 2549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638109 2549 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638113 2549 flags.go:64] FLAG: --cgroup-root="" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638116 2549 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638119 2549 flags.go:64] FLAG: --client-ca-file="" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638122 2549 flags.go:64] FLAG: --cloud-provider="external" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638125 2549 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638128 2549 flags.go:64] FLAG: --cluster-domain="" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638131 2549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638134 2549 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638137 2549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638141 2549 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 14:01:14.643671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638145 2549 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638148 2549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638151 2549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638155 2549 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638158 2549 flags.go:64] FLAG: --contention-profiling="false" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638161 2549 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638165 2549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638168 2549 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638175 2549 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638179 2549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638183 2549 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638186 2549 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638189 2549 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638192 2549 flags.go:64] FLAG: --enable-server="true" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638195 2549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638200 2549 flags.go:64] FLAG: --event-burst="100" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638203 2549 flags.go:64] FLAG: --event-qps="50" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638206 2549 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638210 2549 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638212 2549 flags.go:64] FLAG: --eviction-hard="" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638221 2549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638225 2549 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638228 2549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638231 2549 flags.go:64] FLAG: --eviction-soft="" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638234 2549 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 14:01:14.644300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638237 2549 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638240 2549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638243 2549 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638246 2549 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638249 2549 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638252 2549 flags.go:64] FLAG: --feature-gates="" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638256 2549 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638259 2549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638263 2549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638266 2549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638268 2549 flags.go:64] FLAG: --healthz-port="10248" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638272 2549 flags.go:64] FLAG: --help="false" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638275 2549 flags.go:64] FLAG: --hostname-override="ip-10-0-130-28.ec2.internal" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638278 2549 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638281 2549 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638285 2549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638289 2549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638293 2549 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638295 2549 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638298 2549 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638301 2549 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638304 2549 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638307 2549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638323 2549 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 14:01:14.644948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638326 2549 flags.go:64] FLAG: --kube-reserved="" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638329 2549 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638332 2549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638338 2549 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638341 2549 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638344 2549 flags.go:64] FLAG: --lock-file="" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638347 2549 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638351 2549 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638354 2549 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638359 2549 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638362 2549 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638365 2549 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638368 2549 flags.go:64] FLAG: --logging-format="text" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638371 2549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638374 2549 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638377 2549 flags.go:64] FLAG: --manifest-url="" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638380 2549 flags.go:64] FLAG: --manifest-url-header="" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638385 2549 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638388 2549 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638397 2549 flags.go:64] FLAG: --max-pods="110" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638400 2549 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638403 2549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638406 2549 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638411 2549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638415 2549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 14:01:14.645599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638418 2549 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638421 2549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638428 2549 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638431 2549 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638435 2549 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638438 2549 flags.go:64] FLAG: --pod-cidr="" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638441 2549 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638447 2549 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638449 2549 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638453 2549 flags.go:64] FLAG: --pods-per-core="0" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638456 2549 flags.go:64] FLAG: --port="10250" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638461 2549 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638464 2549 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-054a617f9030cd263" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638468 2549 flags.go:64] FLAG: --qos-reserved="" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638472 2549 flags.go:64] FLAG: --read-only-port="10255" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638475 2549 flags.go:64] FLAG: --register-node="true" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638478 2549 flags.go:64] FLAG: --register-with-taints="" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638482 2549 flags.go:64] FLAG: --registry-burst="10" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638484 2549 flags.go:64] FLAG: --registry-qps="5" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638487 2549 flags.go:64] FLAG: --reserved-cpus="" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638490 2549 flags.go:64] FLAG: --reserved-memory="" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638494 2549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638497 2549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638500 2549 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638503 2549 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 14:01:14.646212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638506 2549 flags.go:64] FLAG: --runonce="false" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638509 2549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638513 2549 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638516 2549 flags.go:64] FLAG: --seccomp-default="false" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638519 2549 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638523 2549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638526 2549 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638530 2549 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638533 2549 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638536 2549 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638539 2549 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638542 2549 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638545 2549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638548 2549 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638550 2549 flags.go:64] FLAG: --system-cgroups="" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638553 2549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638559 2549 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638562 2549 flags.go:64] FLAG: --tls-cert-file="" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638566 2549 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638570 2549 flags.go:64] FLAG: --tls-min-version="" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638573 2549 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638577 2549 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638579 2549 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638582 2549 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638585 2549 flags.go:64] FLAG: --v="2" Jul 29 14:01:14.647220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638589 2549 flags.go:64] FLAG: --version="false" Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638594 2549 flags.go:64] FLAG: --vmodule="" Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638598 2549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.638601 2549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638780 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638785 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638789 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638791 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638794 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638796 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638799 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638802 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638806 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638808 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638811 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638814 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638816 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638819 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638821 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638824 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:14.648022 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638827 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638830 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638832 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638834 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638837 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638842 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638845 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638847 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638850 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638853 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638856 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638859 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638861 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638864 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638867 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638869 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638872 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638876 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638880 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:14.648554 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638883 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638885 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638888 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638890 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638893 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638896 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638900 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638902 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638905 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638908 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638910 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638913 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638915 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638918 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638920 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638923 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638926 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638928 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638933 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638935 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:14.649015 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638938 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638941 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638943 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638948 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638951 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638954 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638957 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638960 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638963 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638966 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638970 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638973 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638976 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638978 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638981 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638984 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638986 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638990 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638992 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638995 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:14.649511 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.638998 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639001 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639003 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639006 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639009 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639011 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639014 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639017 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639019 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639022 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639026 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639029 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639031 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639034 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639036 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639041 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639044 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639047 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639050 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:14.650004 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639053 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639056 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639058 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639062 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.639064 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.639694 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.648394 2549 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.648415 2549 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.648438 2549 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.648445 2549 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648509 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648515 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648520 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648525 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:14.650481 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648528 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648531 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648534 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648537 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648540 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648543 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648545 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648548 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648551 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648553 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648556 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648559 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648562 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648564 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648567 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648570 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648573 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648576 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648579 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:14.650853 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648582 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648584 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648587 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648589 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648592 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648595 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648599 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648601 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648604 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648606 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648609 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648611 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648614 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648617 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648621 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648625 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648628 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648631 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648634 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:14.651337 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648637 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648640 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648643 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648645 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648648 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648651 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648654 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648657 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648659 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648662 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648664 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648667 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648670 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648673 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648675 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648678 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648680 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648683 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648686 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648689 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:14.651819 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648692 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648695 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648698 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648700 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648703 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648706 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648709 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648711 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648715 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648718 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648721 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648724 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648727 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648730 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648733 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648736 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648738 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648742 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648745 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648748 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:14.652418 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648750 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648753 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648756 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648759 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648762 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648764 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648767 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648770 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648773 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648775 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648778 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648781 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648783 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648786 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648789 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648791 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:14.652923 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648794 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.648800 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648925 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648931 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648936 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648941 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648944 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648947 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648950 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648953 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648956 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648959 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648962 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648964 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:14.653319 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648968 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648971 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648974 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648977 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648979 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648983 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648986 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648989 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648992 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648994 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.648997 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649000 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649002 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649004 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649008 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649010 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649013 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649016 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649018 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:14.653666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649022 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649026 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649029 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649032 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649035 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649038 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649040 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649043 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649046 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649049 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649051 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649054 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649056 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649059 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649062 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649065 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649069 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649072 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649076 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649079 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:14.654137 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649081 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649084 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649087 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649090 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649092 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649095 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649097 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649101 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649104 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649106 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649109 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649111 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649114 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649117 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649119 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649122 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649125 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649128 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649130 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649132 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:14.654638 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649135 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649138 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649140 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649143 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649145 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649148 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649150 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649153 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649156 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649159 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649161 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649164 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649167 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649169 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649172 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649175 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649177 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649180 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649182 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649185 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:14.655121 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649188 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649191 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649193 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649196 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649198 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649201 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649203 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:14.649206 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.649212 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.649367 2549 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.653268 2549 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.654475 2549 server.go:1019] "Starting client certificate rotation" Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.654589 2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 14:01:14.655630 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.654632 2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 14:01:14.657028 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.656998 2549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 14:01:14.658984 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.658966 2549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 14:01:14.671651 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.671626 2549 log.go:25] "Validated CRI v1 runtime API" Jul 29 14:01:14.673847 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.673828 2549 log.go:25] "Validated CRI v1 image API" Jul 29 14:01:14.674794 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.674778 2549 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 14:01:14.680927 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.680901 2549 fs.go:135] Filesystem UUIDs: map[5dce878c-4408-4386-89a4-71d2678fdcc8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9a49e0e9-f4a3-4e1d-936d-2b9cef8f4cef:/dev/nvme0n1p3] Jul 29 14:01:14.681025 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.680923 2549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 14:01:14.685283 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.685259 2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 14:01:14.686087 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.685971 2549 manager.go:216] Machine: {Timestamp:2026-07-29 14:01:14.684610518 +0000 UTC m=+0.337204343 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105310 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cce272c96c2a6cde6ab1f9890fad1 SystemUUID:ec2cce27-2c96-c2a6-cde6-ab1f9890fad1 BootID:8f39c76e-067d-456f-8f0b-13849d3f6087 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:50:c0:2c:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:50:c0:2c:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:92:42:89:65:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 14:01:14.686087 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.686087 2549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 14:01:14.686206 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.686192 2549 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 14:01:14.689126 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.689096 2549 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 14:01:14.689355 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.689130 2549 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 14:01:14.690112 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.690085 2549 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 14:01:14.690112 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.690112 2549 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 14:01:14.690293 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.690125 2549 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 14:01:14.691700 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.691663 2549 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 14:01:14.693043 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.693030 2549 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 14:01:14.696426 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.696411 2549 state_mem.go:36] "Initialized new in-memory state store" Jul 29 14:01:14.696569 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.696559 2549 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 14:01:14.696610 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.696592 2549 kubelet.go:486] "Attempting to sync node with API server" Jul 29 14:01:14.696610 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.696603 2549 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 14:01:14.696661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.696616 2549 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 14:01:14.696661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.696626 2549 kubelet.go:398] "Adding apiserver pod source" Jul 29 14:01:14.696661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.696636 2549 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 14:01:14.697073 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.697054 2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 14:01:14.697073 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.697077 2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 14:01:14.699754 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.699734 2549 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 14:01:14.701198 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.701182 2549 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 14:01:14.701248 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.701213 2549 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 14:01:14.701865 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.701854 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 14:01:14.701900 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.701870 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 14:01:14.701900 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.701877 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 14:01:14.701900 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.701883 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 14:01:14.701900 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.701891 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 14:01:14.701900 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.701897 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 14:01:14.702049 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.702037 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 14:01:14.702049 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.702043 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 14:01:14.702107 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.702052 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 14:01:14.702107 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.702058 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 14:01:14.702107 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.702067 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 14:01:14.702107 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.702075 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 14:01:14.703026 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.703015 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 14:01:14.703026 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.703026 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 14:01:14.706064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.706043 2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ml5kx" Jul 29 14:01:14.706628 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.706616 2549 server.go:1268] "Started kubelet" Jul 29 14:01:14.706750 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.706706 2549 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 14:01:14.706866 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.706720 2549 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 14:01:14.706866 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.706774 2549 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 14:01:14.707355 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.707332 2549 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 14:01:14.707442 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.707360 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 14:01:14.707442 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.707361 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 14:01:14.707677 ip-10-0-130-28 systemd[1]: Started Kubernetes Kubelet. Jul 29 14:01:14.708781 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.708704 2549 server.go:310] "Adding debug handlers to kubelet server" Jul 29 14:01:14.712473 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.712411 2549 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 14:01:14.712604 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.711545 2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-28.ec2.internal.18c6c7849e6dcb1d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-28.ec2.internal,UID:ip-10-0-130-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-28.ec2.internal,},FirstTimestamp:2026-07-29 14:01:14.706586397 +0000 UTC m=+0.359180221,LastTimestamp:2026-07-29 14:01:14.706586397 +0000 UTC m=+0.359180221,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-28.ec2.internal,}" Jul 29 14:01:14.713777 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.713756 2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ml5kx" Jul 29 14:01:14.713987 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.713959 2549 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 14:01:14.719057 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.719039 2549 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 14:01:14.719133 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.719064 2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 14:01:14.719275 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.719247 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:14.719751 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.719729 2549 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 14:01:14.719850 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.719754 2549 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 14:01:14.719850 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.719837 2549 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 14:01:14.720302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.720284 2549 factory.go:55] Registering systemd factory Jul 29 14:01:14.720414 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.720305 2549 factory.go:223] Registration of the systemd container factory successfully Jul 29 14:01:14.720496 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.720428 2549 reconstruct.go:97] "Volume reconstruction finished" Jul 29 14:01:14.720496 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.720437 2549 reconciler.go:29] "Reconciler: start to sync state" Jul 29 14:01:14.720622 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.720603 2549 factory.go:153] Registering CRI-O factory Jul 29 14:01:14.720688 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.720627 2549 factory.go:223] Registration of the crio container factory successfully Jul 29 14:01:14.720746 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.720686 2549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 14:01:14.720746 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.720713 2549 factory.go:103] Registering Raw factory Jul 29 14:01:14.720746 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.720730 2549 manager.go:1196] Started watching for new ooms in manager Jul 29 14:01:14.721434 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.721421 2549 manager.go:349] Starting recovery of all containers Jul 29 14:01:14.730775 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.730539 2549 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:14.733741 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.733712 2549 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-28.ec2.internal\" not found" node="ip-10-0-130-28.ec2.internal" Jul 29 14:01:14.733862 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.733844 2549 manager.go:354] Recovery completed Jul 29 14:01:14.738578 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.738560 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:14.740422 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.740405 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:14.740491 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.740439 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:14.740491 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.740450 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:14.741014 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.741000 2549 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 14:01:14.741014 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.741011 2549 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 14:01:14.741135 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.741014 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:14.741135 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.741032 2549 state_mem.go:36] "Initialized new in-memory state store" Jul 29 14:01:14.741878 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.741861 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:14.741973 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.741896 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:14.741973 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.741912 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:14.742456 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.742442 2549 policy_none.go:49] "None policy: Start" Jul 29 14:01:14.742509 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.742463 2549 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 14:01:14.742509 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.742477 2549 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 14:01:14.746910 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.746893 2549 policy_none.go:47] "Start" Jul 29 14:01:14.793065 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.793040 2549 manager.go:337] "Starting Device Plugin manager" Jul 29 14:01:14.793226 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.793091 2549 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 14:01:14.793226 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.793105 2549 server.go:87] "Starting device plugin registration server" Jul 29 14:01:14.793463 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.793447 2549 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 14:01:14.793529 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.793464 2549 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 14:01:14.793603 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.793589 2549 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 14:01:14.794190 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.793745 2549 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 14:01:14.794190 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.793761 2549 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 14:01:14.794401 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.794370 2549 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 14:01:14.794488 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.794475 2549 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:14.856646 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.856598 2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 14:01:14.857943 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.857922 2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 14:01:14.858010 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.857971 2549 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 14:01:14.858010 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.858002 2549 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 14:01:14.858106 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.858046 2549 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 14:01:14.861049 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.861018 2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:14.893852 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.893741 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:14.894905 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.894887 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:14.894996 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.894919 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:14.895380 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.895366 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:14.895474 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.895404 2549 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-28.ec2.internal" Jul 29 14:01:14.903574 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.903554 2549 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-28.ec2.internal" Jul 29 14:01:14.903631 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.903588 2549 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-28.ec2.internal\": node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:14.918031 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.917999 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:14.958808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.958776 2549 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"] Jul 29 14:01:14.958901 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.958873 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:14.960540 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.960519 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:14.960654 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.960558 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:14.960654 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.960581 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:14.961947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.961934 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:14.962066 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.962051 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Jul 29 14:01:14.962105 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.962086 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:14.962773 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.962756 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:14.962853 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.962793 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:14.962853 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.962807 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:14.962853 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.962758 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:14.962956 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.962864 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:14.962956 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.962879 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:14.964087 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.964071 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Jul 29 14:01:14.964136 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.964103 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:14.964927 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.964907 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:14.965026 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.964940 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:14.965026 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:14.964952 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:14.981099 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.981068 2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-28.ec2.internal\" not found" node="ip-10-0-130-28.ec2.internal" Jul 29 14:01:14.984845 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:14.984824 2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-28.ec2.internal\" not found" node="ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.018238 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:15.018203 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:15.021560 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.021534 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/399ee5460b44b20743b00de1e70c4992-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"399ee5460b44b20743b00de1e70c4992\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.021667 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.021572 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa70b4877e9abd3c146a0f6d52419063-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"aa70b4877e9abd3c146a0f6d52419063\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.021667 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.021594 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/399ee5460b44b20743b00de1e70c4992-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"399ee5460b44b20743b00de1e70c4992\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.119214 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:15.119172 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:15.122512 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.122482 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa70b4877e9abd3c146a0f6d52419063-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"aa70b4877e9abd3c146a0f6d52419063\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.122609 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.122526 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/399ee5460b44b20743b00de1e70c4992-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"399ee5460b44b20743b00de1e70c4992\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.122609 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.122546 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/399ee5460b44b20743b00de1e70c4992-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"399ee5460b44b20743b00de1e70c4992\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.122609 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.122589 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/399ee5460b44b20743b00de1e70c4992-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"399ee5460b44b20743b00de1e70c4992\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.122710 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.122588 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa70b4877e9abd3c146a0f6d52419063-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"aa70b4877e9abd3c146a0f6d52419063\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.122710 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.122593 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/399ee5460b44b20743b00de1e70c4992-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"399ee5460b44b20743b00de1e70c4992\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.220003 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:15.219924 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:15.283296 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.283269 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.288714 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.288695 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.320989 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:15.320951 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:15.421496 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:15.421439 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:15.521927 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:15.521891 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:15.622457 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:15.622422 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:15.653717 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.653681 2549 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 14:01:15.654422 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.653849 2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 14:01:15.654422 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.653893 2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 14:01:15.719206 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.718681 2549 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:15.719206 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.718595 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 13:56:15 +0000 UTC" deadline="2028-03-03 16:05:34.863637018 +0000 UTC" Jul 29 14:01:15.719206 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.718852 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13994h4m19.144795513s" Jul 29 14:01:15.722652 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:15.722623 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Jul 29 14:01:15.801076 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:15.801042 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399ee5460b44b20743b00de1e70c4992.slice/crio-4029854a317e36ad53646d885e75c6a99f7c37c0fe4dcff4cd1e32e706901562 WatchSource:0}: Error finding container 4029854a317e36ad53646d885e75c6a99f7c37c0fe4dcff4cd1e32e706901562: Status 404 returned error can't find the container with id 4029854a317e36ad53646d885e75c6a99f7c37c0fe4dcff4cd1e32e706901562 Jul 29 14:01:15.803924 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:15.803899 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa70b4877e9abd3c146a0f6d52419063.slice/crio-cf53206e21974d55a3f75cd3249158ead91d49be6a188a4053f7a7b29b9659e4 WatchSource:0}: Error finding container cf53206e21974d55a3f75cd3249158ead91d49be6a188a4053f7a7b29b9659e4: Status 404 returned error can't find the container with id cf53206e21974d55a3f75cd3249158ead91d49be6a188a4053f7a7b29b9659e4 Jul 29 14:01:15.809243 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.808652 2549 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:15.810189 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.810141 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:01:15.820396 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.820368 2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.830742 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.830718 2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 14:01:15.831751 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.831734 2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Jul 29 14:01:15.840216 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.840194 2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 14:01:15.861686 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.861637 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" event={"ID":"aa70b4877e9abd3c146a0f6d52419063","Type":"ContainerStarted","Data":"cf53206e21974d55a3f75cd3249158ead91d49be6a188a4053f7a7b29b9659e4"} Jul 29 14:01:15.862656 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:15.862634 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"399ee5460b44b20743b00de1e70c4992","Type":"ContainerStarted","Data":"4029854a317e36ad53646d885e75c6a99f7c37c0fe4dcff4cd1e32e706901562"} Jul 29 14:01:16.600704 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.600605 2549 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:16.697960 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.697911 2549 apiserver.go:52] "Watching apiserver" Jul 29 14:01:16.706600 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.706562 2549 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 14:01:16.707135 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.707110 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-dwzkb","kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal","openshift-cluster-node-tuning-operator/tuned-twwwt","openshift-network-diagnostics/network-check-target-2cg6g","openshift-network-operator/iptables-alerter-st7fm","openshift-ovn-kubernetes/ovnkube-node-qnvb6","kube-system/konnectivity-agent-8dml7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s","openshift-dns/node-resolver-wdzkw","openshift-image-registry/node-ca-vfwlj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal","openshift-multus/multus-8czqh","openshift-multus/multus-additional-cni-plugins-chbrj","openshift-multus/network-metrics-daemon-nd944"] Jul 29 14:01:16.709098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.709065 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:16.712340 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.711990 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.713627 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.713209 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4dnrb\"" Jul 29 14:01:16.713627 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.713369 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 14:01:16.713627 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.713523 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 14:01:16.715210 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.714525 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.717070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.716236 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 14:01:16.717070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.716478 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ccv6q\"" Jul 29 14:01:16.717070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.716767 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 14:01:16.717070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.716985 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 14:01:16.717070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.717037 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:01:16.717430 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.717167 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 14:01:16.717759 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.717497 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:16.717759 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.717568 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bpbqm\"" Jul 29 14:01:16.717759 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.717595 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:16.717759 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.717696 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 14:01:16.718071 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.717839 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 14:01:16.718420 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.718383 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.719347 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.719321 2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 14:01:16.721103 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.720721 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.721728 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.721709 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 14:01:16.721825 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.721772 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-78lgq\"" Jul 29 14:01:16.721825 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.721708 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:01:16.721943 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.721879 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 14:01:16.722164 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.722138 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.723433 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.723409 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xgc9t\"" Jul 29 14:01:16.723652 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.723635 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 14:01:16.724416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.724379 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 14:01:16.724509 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.724424 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 14:01:16.724687 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.724669 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:16.724802 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.724777 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:16.725125 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.725106 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 14:01:16.725196 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.725156 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 14:01:16.725285 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.725106 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 14:01:16.725536 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.725515 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 14:01:16.725668 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.725634 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sgq4z\"" Jul 29 14:01:16.725753 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.725728 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 14:01:16.726515 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.726493 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.730297 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730268 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-slash\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.730439 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730311 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-var-lib-openvswitch\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.730439 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730365 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-tuned\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.730439 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730420 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dd2825-35bc-4921-9b52-456e3a5df0f6-tmp\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.730687 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730458 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33b01cb7-6fc6-4029-8054-bdd9a1043def-iptables-alerter-script\") pod \"iptables-alerter-st7fm\" (UID: \"33b01cb7-6fc6-4029-8054-bdd9a1043def\") " pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.730687 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730494 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-run-ovn\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.730687 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730521 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-cni-bin\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.730687 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730554 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83f582eb-331b-4728-9e63-a5c94499d8b3-env-overrides\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.730687 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730589 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-sysconfig\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.730687 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730619 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-sys\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.730687 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730646 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-host\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.731028 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730707 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e3ccd9f-be11-45b0-8380-fe38fdd54579-hosts-file\") pod \"node-resolver-wdzkw\" (UID: \"7e3ccd9f-be11-45b0-8380-fe38fdd54579\") " pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.731028 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730742 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-systemd-units\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.731028 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730781 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mq9\" (UniqueName: \"kubernetes.io/projected/83f582eb-331b-4728-9e63-a5c94499d8b3-kube-api-access-s9mq9\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.731028 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.730978 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-sysctl-conf\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.731028 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731012 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-lib-modules\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.731269 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731039 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33b01cb7-6fc6-4029-8054-bdd9a1043def-host-slash\") pod \"iptables-alerter-st7fm\" (UID: \"33b01cb7-6fc6-4029-8054-bdd9a1043def\") " pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.731269 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731087 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/06d8cf0b-e7fd-4c62-a798-c6e26bda4526-agent-certs\") pod \"konnectivity-agent-8dml7\" (UID: \"06d8cf0b-e7fd-4c62-a798-c6e26bda4526\") " pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:16.731369 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731124 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-system-cni-dir\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.731450 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731362 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-run-systemd\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.731450 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731412 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-cni-netd\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.731554 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731457 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83f582eb-331b-4728-9e63-a5c94499d8b3-ovnkube-script-lib\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.731554 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731488 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-systemd\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.731554 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731523 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5c29\" (UniqueName: \"kubernetes.io/projected/07dd2825-35bc-4921-9b52-456e3a5df0f6-kube-api-access-n5c29\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.731696 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731560 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrc5q\" (UniqueName: \"kubernetes.io/projected/33b01cb7-6fc6-4029-8054-bdd9a1043def-kube-api-access-xrc5q\") pod \"iptables-alerter-st7fm\" (UID: \"33b01cb7-6fc6-4029-8054-bdd9a1043def\") " pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.731696 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731598 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.731696 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731637 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/618f9616-1888-442e-879b-1739233920c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.731839 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731767 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-etc-openvswitch\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.732017 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.731994 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-node-log\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.732079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732042 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.732133 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732087 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-sysctl-d\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.732188 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732138 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-var-lib-kubelet\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.732245 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732185 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3ccd9f-be11-45b0-8380-fe38fdd54579-tmp-dir\") pod \"node-resolver-wdzkw\" (UID: \"7e3ccd9f-be11-45b0-8380-fe38fdd54579\") " pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.732245 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732222 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/06d8cf0b-e7fd-4c62-a798-c6e26bda4526-konnectivity-ca\") pod \"konnectivity-agent-8dml7\" (UID: \"06d8cf0b-e7fd-4c62-a798-c6e26bda4526\") " pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:16.732342 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732271 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/618f9616-1888-442e-879b-1739233920c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.732342 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732314 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-run-openvswitch\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.732488 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732356 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-log-socket\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.732488 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732420 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-modprobe-d\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.732488 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732449 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44z46\" (UniqueName: \"kubernetes.io/projected/7e3ccd9f-be11-45b0-8380-fe38fdd54579-kube-api-access-44z46\") pod \"node-resolver-wdzkw\" (UID: \"7e3ccd9f-be11-45b0-8380-fe38fdd54579\") " pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.732488 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732478 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/618f9616-1888-442e-879b-1739233920c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.732684 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732511 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnl6b\" (UniqueName: \"kubernetes.io/projected/618f9616-1888-442e-879b-1739233920c5-kube-api-access-fnl6b\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.732684 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732539 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:16.732684 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732568 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83f582eb-331b-4728-9e63-a5c94499d8b3-ovn-node-metrics-cert\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.732684 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732642 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-kubernetes\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.732884 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732686 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-cnibin\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.732884 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732731 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-kubelet\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.732884 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732767 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-run-netns\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.732884 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732822 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.732884 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732854 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83f582eb-331b-4728-9e63-a5c94499d8b3-ovnkube-config\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.733132 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732902 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-run\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.733132 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.732939 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-os-release\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.733704 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.733660 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.734052 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.734029 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w7hfg\"" Jul 29 14:01:16.736909 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.736868 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 14:01:16.737334 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.737074 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 14:01:16.737334 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.737102 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 14:01:16.737334 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.737174 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 14:01:16.737334 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.737196 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 14:01:16.737334 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.737109 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qck8x\"" Jul 29 14:01:16.737655 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.737545 2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 14:01:16.737984 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.737809 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 14:01:16.739543 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.738969 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.741602 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.741033 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:16.741602 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.741144 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:16.742245 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.742067 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 14:01:16.742245 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.742181 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kwvxs\"" Jul 29 14:01:16.756364 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.756333 2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xr4vt" Jul 29 14:01:16.762313 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.762284 2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xr4vt" Jul 29 14:01:16.821614 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.821578 2549 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 14:01:16.833542 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833501 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-cnibin\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.833808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833550 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-run-openvswitch\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.833808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833584 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-log-socket\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.833808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833649 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-log-socket\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.833808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833657 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-run-openvswitch\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.833808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833694 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-modprobe-d\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.833808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833735 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/618f9616-1888-442e-879b-1739233920c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.833808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833767 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-registration-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.833808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833792 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-sys-fs\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833819 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7zp9\" (UniqueName: \"kubernetes.io/projected/d78ddb82-d84a-4565-8bc9-018d47f5a549-kube-api-access-v7zp9\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833843 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436-serviceca\") pod \"node-ca-vfwlj\" (UID: \"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436\") " pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833871 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-modprobe-d\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833873 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-cnibin\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833915 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-etc-kubernetes\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833927 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-cnibin\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.833968 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba453020-a7c0-4204-988e-cfc912eafadb-kubelet-config\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834005 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-kubelet\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834042 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-run-netns\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834065 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83f582eb-331b-4728-9e63-a5c94499d8b3-ovnkube-config\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834092 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-run\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834116 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-os-release\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834168 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-var-lib-kubelet\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834192 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dd2825-35bc-4921-9b52-456e3a5df0f6-tmp\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.834232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834219 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-cni-bin\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834246 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83f582eb-331b-4728-9e63-a5c94499d8b3-env-overrides\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834267 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-sysconfig\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834287 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-host\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834304 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e3ccd9f-be11-45b0-8380-fe38fdd54579-hosts-file\") pod \"node-resolver-wdzkw\" (UID: \"7e3ccd9f-be11-45b0-8380-fe38fdd54579\") " pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834323 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-var-lib-cni-multus\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834343 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-run-multus-certs\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834376 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-systemd-units\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834403 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mq9\" (UniqueName: \"kubernetes.io/projected/83f582eb-331b-4728-9e63-a5c94499d8b3-kube-api-access-s9mq9\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834421 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-sysctl-conf\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834443 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-lib-modules\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834482 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33b01cb7-6fc6-4029-8054-bdd9a1043def-host-slash\") pod \"iptables-alerter-st7fm\" (UID: \"33b01cb7-6fc6-4029-8054-bdd9a1043def\") " pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834502 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-system-cni-dir\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834519 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834535 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-etc-selinux\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834552 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-systemd\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834569 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5c29\" (UniqueName: \"kubernetes.io/projected/07dd2825-35bc-4921-9b52-456e3a5df0f6-kube-api-access-n5c29\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.835004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834587 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrc5q\" (UniqueName: \"kubernetes.io/projected/33b01cb7-6fc6-4029-8054-bdd9a1043def-kube-api-access-xrc5q\") pod \"iptables-alerter-st7fm\" (UID: \"33b01cb7-6fc6-4029-8054-bdd9a1043def\") " pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834614 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834663 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/618f9616-1888-442e-879b-1739233920c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834668 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/618f9616-1888-442e-879b-1739233920c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834694 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/618f9616-1888-442e-879b-1739233920c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834717 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-socket-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834734 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af643c4d-5e6e-4ced-a948-653326504d7f-cni-binary-copy\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834759 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/06d8cf0b-e7fd-4c62-a798-c6e26bda4526-konnectivity-ca\") pod \"konnectivity-agent-8dml7\" (UID: \"06d8cf0b-e7fd-4c62-a798-c6e26bda4526\") " pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834798 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834834 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44z46\" (UniqueName: \"kubernetes.io/projected/7e3ccd9f-be11-45b0-8380-fe38fdd54579-kube-api-access-44z46\") pod \"node-resolver-wdzkw\" (UID: \"7e3ccd9f-be11-45b0-8380-fe38fdd54579\") " pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834864 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnl6b\" (UniqueName: \"kubernetes.io/projected/618f9616-1888-442e-879b-1739233920c5-kube-api-access-fnl6b\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834893 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834922 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-system-cni-dir\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834950 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-hostroot\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.834978 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thqk\" (UniqueName: \"kubernetes.io/projected/af643c4d-5e6e-4ced-a948-653326504d7f-kube-api-access-8thqk\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835005 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpnq5\" (UniqueName: \"kubernetes.io/projected/4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436-kube-api-access-lpnq5\") pod \"node-ca-vfwlj\" (UID: \"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436\") " pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.835861 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835032 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba453020-a7c0-4204-988e-cfc912eafadb-dbus\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835061 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83f582eb-331b-4728-9e63-a5c94499d8b3-ovn-node-metrics-cert\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835089 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-kubernetes\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835119 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af643c4d-5e6e-4ced-a948-653326504d7f-multus-daemon-config\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835143 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436-host\") pod \"node-ca-vfwlj\" (UID: \"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436\") " pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835172 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835202 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5n9\" (UniqueName: \"kubernetes.io/projected/ce6e0886-1615-4821-bba0-9cdba134b1d4-kube-api-access-ch5n9\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835231 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-run-k8s-cni-cncf-io\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835253 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/06d8cf0b-e7fd-4c62-a798-c6e26bda4526-konnectivity-ca\") pod \"konnectivity-agent-8dml7\" (UID: \"06d8cf0b-e7fd-4c62-a798-c6e26bda4526\") " pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835267 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-slash\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835318 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-slash\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835322 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-kubelet\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.835343 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-run-netns\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836045 2549 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836154 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-sysctl-conf\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836198 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83f582eb-331b-4728-9e63-a5c94499d8b3-ovnkube-config\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836224 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-run\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836281 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33b01cb7-6fc6-4029-8054-bdd9a1043def-host-slash\") pod \"iptables-alerter-st7fm\" (UID: \"33b01cb7-6fc6-4029-8054-bdd9a1043def\") " pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.836662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836291 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-os-release\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836453 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836585 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-lib-modules\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836591 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/618f9616-1888-442e-879b-1739233920c5-system-cni-dir\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.836682 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-systemd\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837058 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/618f9616-1888-442e-879b-1739233920c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837246 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837299 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-kubernetes\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837380 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-var-lib-openvswitch\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837418 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-tuned\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837442 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33b01cb7-6fc6-4029-8054-bdd9a1043def-iptables-alerter-script\") pod \"iptables-alerter-st7fm\" (UID: \"33b01cb7-6fc6-4029-8054-bdd9a1043def\") " pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837458 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/618f9616-1888-442e-879b-1739233920c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837465 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-run-ovn\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837492 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-run-ovn\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837519 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-sys\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837548 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/06d8cf0b-e7fd-4c62-a798-c6e26bda4526-agent-certs\") pod \"konnectivity-agent-8dml7\" (UID: \"06d8cf0b-e7fd-4c62-a798-c6e26bda4526\") " pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:16.837585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837581 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-device-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837592 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-cni-bin\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837608 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-multus-socket-dir-parent\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837636 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-var-lib-cni-bin\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837880 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-systemd-units\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837925 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e3ccd9f-be11-45b0-8380-fe38fdd54579-hosts-file\") pod \"node-resolver-wdzkw\" (UID: \"7e3ccd9f-be11-45b0-8380-fe38fdd54579\") " pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837964 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-sysconfig\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.837991 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-host\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838020 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-run-systemd\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838038 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83f582eb-331b-4728-9e63-a5c94499d8b3-env-overrides\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838049 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-cni-netd\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838075 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83f582eb-331b-4728-9e63-a5c94499d8b3-ovnkube-script-lib\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838102 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-run-systemd\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838156 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-cni-netd\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838155 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-sys\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838199 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-var-lib-openvswitch\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838105 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838249 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-os-release\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.838406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838407 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-run-netns\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838441 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-multus-conf-dir\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838484 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-etc-openvswitch\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838572 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-etc-openvswitch\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.838828 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33b01cb7-6fc6-4029-8054-bdd9a1043def-iptables-alerter-script\") pod \"iptables-alerter-st7fm\" (UID: \"33b01cb7-6fc6-4029-8054-bdd9a1043def\") " pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839027 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-node-log\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839060 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839086 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-sysctl-d\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839112 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-var-lib-kubelet\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839127 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83f582eb-331b-4728-9e63-a5c94499d8b3-ovnkube-script-lib\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839137 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3ccd9f-be11-45b0-8380-fe38fdd54579-tmp-dir\") pod \"node-resolver-wdzkw\" (UID: \"7e3ccd9f-be11-45b0-8380-fe38fdd54579\") " pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839167 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-multus-cni-dir\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839179 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839300 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83f582eb-331b-4728-9e63-a5c94499d8b3-node-log\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.839416 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839369 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-var-lib-kubelet\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.840012 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839455 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-sysctl-d\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.840012 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.839546 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3ccd9f-be11-45b0-8380-fe38fdd54579-tmp-dir\") pod \"node-resolver-wdzkw\" (UID: \"7e3ccd9f-be11-45b0-8380-fe38fdd54579\") " pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.841040 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.841015 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dd2825-35bc-4921-9b52-456e3a5df0f6-tmp\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.841182 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.841112 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07dd2825-35bc-4921-9b52-456e3a5df0f6-etc-tuned\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.842718 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.842695 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83f582eb-331b-4728-9e63-a5c94499d8b3-ovn-node-metrics-cert\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.842829 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.842747 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/06d8cf0b-e7fd-4c62-a798-c6e26bda4526-agent-certs\") pod \"konnectivity-agent-8dml7\" (UID: \"06d8cf0b-e7fd-4c62-a798-c6e26bda4526\") " pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:16.843058 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.843035 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:16.843147 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.843063 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:16.843147 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.843079 2549 projected.go:196] Error preparing data for projected volume kube-api-access-7pwm5 for pod openshift-network-diagnostics/network-check-target-2cg6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:16.843275 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.843148 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5 podName:4cf66c81-3391-4691-8c9b-485548ca7061 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:17.343124995 +0000 UTC m=+2.995718827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7pwm5" (UniqueName: "kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5") pod "network-check-target-2cg6g" (UID: "4cf66c81-3391-4691-8c9b-485548ca7061") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:16.845798 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.845727 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrc5q\" (UniqueName: \"kubernetes.io/projected/33b01cb7-6fc6-4029-8054-bdd9a1043def-kube-api-access-xrc5q\") pod \"iptables-alerter-st7fm\" (UID: \"33b01cb7-6fc6-4029-8054-bdd9a1043def\") " pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:16.846884 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.846112 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44z46\" (UniqueName: \"kubernetes.io/projected/7e3ccd9f-be11-45b0-8380-fe38fdd54579-kube-api-access-44z46\") pod \"node-resolver-wdzkw\" (UID: \"7e3ccd9f-be11-45b0-8380-fe38fdd54579\") " pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:16.848158 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.848132 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5c29\" (UniqueName: \"kubernetes.io/projected/07dd2825-35bc-4921-9b52-456e3a5df0f6-kube-api-access-n5c29\") pod \"tuned-twwwt\" (UID: \"07dd2825-35bc-4921-9b52-456e3a5df0f6\") " pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:16.849742 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.849714 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mq9\" (UniqueName: \"kubernetes.io/projected/83f582eb-331b-4728-9e63-a5c94499d8b3-kube-api-access-s9mq9\") pod \"ovnkube-node-qnvb6\" (UID: \"83f582eb-331b-4728-9e63-a5c94499d8b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:16.850490 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.850376 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnl6b\" (UniqueName: \"kubernetes.io/projected/618f9616-1888-442e-879b-1739233920c5-kube-api-access-fnl6b\") pod \"multus-additional-cni-plugins-chbrj\" (UID: \"618f9616-1888-442e-879b-1739233920c5\") " pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939559 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939619 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-os-release\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939646 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-run-netns\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939672 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-multus-conf-dir\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939699 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-multus-cni-dir\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.939704 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939725 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-cnibin\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939751 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-registration-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939774 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-sys-fs\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939777 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-os-release\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.939795 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs podName:ce6e0886-1615-4821-bba0-9cdba134b1d4 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:17.439769342 +0000 UTC m=+3.092363170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs") pod "network-metrics-daemon-nd944" (UID: "ce6e0886-1615-4821-bba0-9cdba134b1d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939833 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-sys-fs\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939861 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7zp9\" (UniqueName: \"kubernetes.io/projected/d78ddb82-d84a-4565-8bc9-018d47f5a549-kube-api-access-v7zp9\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939876 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-run-netns\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939891 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436-serviceca\") pod \"node-ca-vfwlj\" (UID: \"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436\") " pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939909 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-multus-conf-dir\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.941224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939922 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-etc-kubernetes\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939951 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba453020-a7c0-4204-988e-cfc912eafadb-kubelet-config\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939965 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-multus-cni-dir\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.939982 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-var-lib-kubelet\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940012 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-cnibin\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940016 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-var-lib-cni-multus\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940043 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-run-multus-certs\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940062 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-registration-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940066 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-etc-kubernetes\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940075 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940101 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-etc-selinux\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940134 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-socket-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940159 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af643c4d-5e6e-4ced-a948-653326504d7f-cni-binary-copy\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940187 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940233 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-system-cni-dir\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940258 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-hostroot\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940284 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thqk\" (UniqueName: \"kubernetes.io/projected/af643c4d-5e6e-4ced-a948-653326504d7f-kube-api-access-8thqk\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940312 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpnq5\" (UniqueName: \"kubernetes.io/projected/4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436-kube-api-access-lpnq5\") pod \"node-ca-vfwlj\" (UID: \"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436\") " pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.942195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940337 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba453020-a7c0-4204-988e-cfc912eafadb-dbus\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940475 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-socket-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940519 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af643c4d-5e6e-4ced-a948-653326504d7f-multus-daemon-config\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940540 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba453020-a7c0-4204-988e-cfc912eafadb-dbus\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940551 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436-host\") pod \"node-ca-vfwlj\" (UID: \"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436\") " pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940581 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5n9\" (UniqueName: \"kubernetes.io/projected/ce6e0886-1615-4821-bba0-9cdba134b1d4-kube-api-access-ch5n9\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940609 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-run-k8s-cni-cncf-io\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940626 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436-serviceca\") pod \"node-ca-vfwlj\" (UID: \"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436\") " pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940646 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-device-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940673 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-multus-socket-dir-parent\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940698 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-var-lib-cni-bin\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940770 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-var-lib-cni-bin\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940823 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba453020-a7c0-4204-988e-cfc912eafadb-kubelet-config\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940826 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436-host\") pod \"node-ca-vfwlj\" (UID: \"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436\") " pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.940648 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:16.940914 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret podName:ba453020-a7c0-4204-988e-cfc912eafadb nodeName:}" failed. No retries permitted until 2026-07-29 14:01:17.440895787 +0000 UTC m=+3.093489616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret") pod "global-pull-secret-syncer-dwzkb" (UID: "ba453020-a7c0-4204-988e-cfc912eafadb") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940929 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-run-multus-certs\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.943208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940927 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-device-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940942 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-multus-socket-dir-parent\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940957 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-var-lib-kubelet\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940944 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-hostroot\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.940975 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-var-lib-cni-multus\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.941005 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-system-cni-dir\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.941010 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.941048 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af643c4d-5e6e-4ced-a948-653326504d7f-host-run-k8s-cni-cncf-io\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.941052 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d78ddb82-d84a-4565-8bc9-018d47f5a549-etc-selinux\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.941414 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af643c4d-5e6e-4ced-a948-653326504d7f-cni-binary-copy\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.944174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.941515 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af643c4d-5e6e-4ced-a948-653326504d7f-multus-daemon-config\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:16.954255 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.952860 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7zp9\" (UniqueName: \"kubernetes.io/projected/d78ddb82-d84a-4565-8bc9-018d47f5a549-kube-api-access-v7zp9\") pod \"aws-ebs-csi-driver-node-9dh4s\" (UID: \"d78ddb82-d84a-4565-8bc9-018d47f5a549\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:16.954255 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.953413 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpnq5\" (UniqueName: \"kubernetes.io/projected/4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436-kube-api-access-lpnq5\") pod \"node-ca-vfwlj\" (UID: \"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436\") " pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:16.954255 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.954196 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5n9\" (UniqueName: \"kubernetes.io/projected/ce6e0886-1615-4821-bba0-9cdba134b1d4-kube-api-access-ch5n9\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:16.955448 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:16.955417 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thqk\" (UniqueName: \"kubernetes.io/projected/af643c4d-5e6e-4ced-a948-653326504d7f-kube-api-access-8thqk\") pod \"multus-8czqh\" (UID: \"af643c4d-5e6e-4ced-a948-653326504d7f\") " pod="openshift-multus/multus-8czqh" Jul 29 14:01:17.027411 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.027358 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:17.038890 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.038844 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-chbrj" Jul 29 14:01:17.050410 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.050368 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-twwwt" Jul 29 14:01:17.060974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.060928 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-st7fm" Jul 29 14:01:17.070197 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.070160 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:17.078484 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.078452 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdzkw" Jul 29 14:01:17.087760 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.087726 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" Jul 29 14:01:17.095831 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.095794 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vfwlj" Jul 29 14:01:17.104122 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.104092 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8czqh" Jul 29 14:01:17.197293 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.197186 2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:17.344279 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.344239 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:17.344498 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.344422 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:17.344498 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.344453 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:17.344498 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.344467 2549 projected.go:196] Error preparing data for projected volume kube-api-access-7pwm5 for pod openshift-network-diagnostics/network-check-target-2cg6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:17.344663 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.344547 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5 podName:4cf66c81-3391-4691-8c9b-485548ca7061 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:18.344524651 +0000 UTC m=+3.997118465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pwm5" (UniqueName: "kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5") pod "network-check-target-2cg6g" (UID: "4cf66c81-3391-4691-8c9b-485548ca7061") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:17.445394 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.445353 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:17.445595 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.445422 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:17.445595 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.445527 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:17.445595 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.445528 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:17.445595 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.445590 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret podName:ba453020-a7c0-4204-988e-cfc912eafadb nodeName:}" failed. No retries permitted until 2026-07-29 14:01:18.445572672 +0000 UTC m=+4.098166489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret") pod "global-pull-secret-syncer-dwzkb" (UID: "ba453020-a7c0-4204-988e-cfc912eafadb") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:17.445792 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.445606 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs podName:ce6e0886-1615-4821-bba0-9cdba134b1d4 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:18.445598941 +0000 UTC m=+4.098192754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs") pod "network-metrics-daemon-nd944" (UID: "ce6e0886-1615-4821-bba0-9cdba134b1d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:17.765133 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.765078 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 13:56:17 +0000 UTC" deadline="2028-04-12 06:52:10.689438822 +0000 UTC" Jul 29 14:01:17.765133 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.765126 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14944h50m52.92431734s" Jul 29 14:01:17.859158 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.859114 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:17.859368 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:17.859293 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:17.859446 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.859408 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:17.859446 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:17.859289 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:18.055719 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:18.055687 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78ddb82_d84a_4565_8bc9_018d47f5a549.slice/crio-eece9ce299c1d7228e59624b9a09faf4a3511f9f8030a87414781f01ad6cd946 WatchSource:0}: Error finding container eece9ce299c1d7228e59624b9a09faf4a3511f9f8030a87414781f01ad6cd946: Status 404 returned error can't find the container with id eece9ce299c1d7228e59624b9a09faf4a3511f9f8030a87414781f01ad6cd946 Jul 29 14:01:18.058556 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:18.058345 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f582eb_331b_4728_9e63_a5c94499d8b3.slice/crio-59007461bb55025f19305a05922161aafef0ef9f3b6997613e3ebc023abe3a93 WatchSource:0}: Error finding container 59007461bb55025f19305a05922161aafef0ef9f3b6997613e3ebc023abe3a93: Status 404 returned error can't find the container with id 59007461bb55025f19305a05922161aafef0ef9f3b6997613e3ebc023abe3a93 Jul 29 14:01:18.062320 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:18.062288 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b01cb7_6fc6_4029_8054_bdd9a1043def.slice/crio-607a790ca0866fe3a7054ef534419d69a7c1ef33a7bd9da87e0cf332dae4dcd9 WatchSource:0}: Error finding container 607a790ca0866fe3a7054ef534419d69a7c1ef33a7bd9da87e0cf332dae4dcd9: Status 404 returned error can't find the container with id 607a790ca0866fe3a7054ef534419d69a7c1ef33a7bd9da87e0cf332dae4dcd9 Jul 29 14:01:18.063378 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:18.063347 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf643c4d_5e6e_4ced_a948_653326504d7f.slice/crio-bc8dc8d96888558d397c46c469b28461b87408ab6a2d904d560670dd01652e2c WatchSource:0}: Error finding container bc8dc8d96888558d397c46c469b28461b87408ab6a2d904d560670dd01652e2c: Status 404 returned error can't find the container with id bc8dc8d96888558d397c46c469b28461b87408ab6a2d904d560670dd01652e2c Jul 29 14:01:18.064814 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:18.064670 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3ccd9f_be11_45b0_8380_fe38fdd54579.slice/crio-00deb0da11e2732369d982c9823fc63fd449e77218d141fb7593338c1a504f76 WatchSource:0}: Error finding container 00deb0da11e2732369d982c9823fc63fd449e77218d141fb7593338c1a504f76: Status 404 returned error can't find the container with id 00deb0da11e2732369d982c9823fc63fd449e77218d141fb7593338c1a504f76 Jul 29 14:01:18.066483 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:18.066288 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8ecdc2_2c1a_47a7_b9a0_97439a6d3436.slice/crio-6256d191e47ed031b7706746b32dcd1bb2cd6c8f12b17b9ff53ead0b377ee88a WatchSource:0}: Error finding container 6256d191e47ed031b7706746b32dcd1bb2cd6c8f12b17b9ff53ead0b377ee88a: Status 404 returned error can't find the container with id 6256d191e47ed031b7706746b32dcd1bb2cd6c8f12b17b9ff53ead0b377ee88a Jul 29 14:01:18.067974 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:18.067949 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07dd2825_35bc_4921_9b52_456e3a5df0f6.slice/crio-a76947c22de50b319be2259dc1e55e915e976993bbd61b51091ad8c024e48f12 WatchSource:0}: Error finding container a76947c22de50b319be2259dc1e55e915e976993bbd61b51091ad8c024e48f12: Status 404 returned error can't find the container with id a76947c22de50b319be2259dc1e55e915e976993bbd61b51091ad8c024e48f12 Jul 29 14:01:18.068615 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:18.068477 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618f9616_1888_442e_879b_1739233920c5.slice/crio-5d86b39bd117d51d9eb1244d1cf2c95f3e6b038c09845db67dbaf43620d17dd8 WatchSource:0}: Error finding container 5d86b39bd117d51d9eb1244d1cf2c95f3e6b038c09845db67dbaf43620d17dd8: Status 404 returned error can't find the container with id 5d86b39bd117d51d9eb1244d1cf2c95f3e6b038c09845db67dbaf43620d17dd8 Jul 29 14:01:18.069786 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:18.069751 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d8cf0b_e7fd_4c62_a798_c6e26bda4526.slice/crio-91182be32d6f7685fba4dd781876df4e9c7bcbfc1ad2b0d0c22271a753a34a15 WatchSource:0}: Error finding container 91182be32d6f7685fba4dd781876df4e9c7bcbfc1ad2b0d0c22271a753a34a15: Status 404 returned error can't find the container with id 91182be32d6f7685fba4dd781876df4e9c7bcbfc1ad2b0d0c22271a753a34a15 Jul 29 14:01:18.354638 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.354594 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:18.354782 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:18.354743 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:18.354782 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:18.354762 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:18.354782 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:18.354771 2549 projected.go:196] Error preparing data for projected volume kube-api-access-7pwm5 for pod openshift-network-diagnostics/network-check-target-2cg6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:18.354908 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:18.354829 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5 podName:4cf66c81-3391-4691-8c9b-485548ca7061 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:20.354812686 +0000 UTC m=+6.007406500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pwm5" (UniqueName: "kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5") pod "network-check-target-2cg6g" (UID: "4cf66c81-3391-4691-8c9b-485548ca7061") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:18.455753 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.455709 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:18.455959 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.455789 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:18.455959 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:18.455895 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:18.456100 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:18.455982 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:18.456100 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:18.455996 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs podName:ce6e0886-1615-4821-bba0-9cdba134b1d4 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:20.455971023 +0000 UTC m=+6.108564856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs") pod "network-metrics-daemon-nd944" (UID: "ce6e0886-1615-4821-bba0-9cdba134b1d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:18.456100 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:18.456073 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret podName:ba453020-a7c0-4204-988e-cfc912eafadb nodeName:}" failed. No retries permitted until 2026-07-29 14:01:20.45605363 +0000 UTC m=+6.108647455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret") pod "global-pull-secret-syncer-dwzkb" (UID: "ba453020-a7c0-4204-988e-cfc912eafadb") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:18.765848 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.765764 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 13:56:17 +0000 UTC" deadline="2028-03-18 14:00:56.254077582 +0000 UTC" Jul 29 14:01:18.765848 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.765845 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14351h59m37.488238171s" Jul 29 14:01:18.863489 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.863086 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:18.863489 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:18.863260 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:18.877455 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.877026 2549 generic.go:356] "Generic (PLEG): container finished" podID="399ee5460b44b20743b00de1e70c4992" containerID="385a3564b5c23befdd696f362453d46f4fd2eb42429b65481478d26ae357aa8c" exitCode=0 Jul 29 14:01:18.877455 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.877184 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"399ee5460b44b20743b00de1e70c4992","Type":"ContainerDied","Data":"385a3564b5c23befdd696f362453d46f4fd2eb42429b65481478d26ae357aa8c"} Jul 29 14:01:18.883103 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.883060 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8dml7" event={"ID":"06d8cf0b-e7fd-4c62-a798-c6e26bda4526","Type":"ContainerStarted","Data":"91182be32d6f7685fba4dd781876df4e9c7bcbfc1ad2b0d0c22271a753a34a15"} Jul 29 14:01:18.896971 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.896933 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerStarted","Data":"5d86b39bd117d51d9eb1244d1cf2c95f3e6b038c09845db67dbaf43620d17dd8"} Jul 29 14:01:18.911756 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.911657 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-twwwt" event={"ID":"07dd2825-35bc-4921-9b52-456e3a5df0f6","Type":"ContainerStarted","Data":"a76947c22de50b319be2259dc1e55e915e976993bbd61b51091ad8c024e48f12"} Jul 29 14:01:18.926953 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.926835 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vfwlj" event={"ID":"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436","Type":"ContainerStarted","Data":"6256d191e47ed031b7706746b32dcd1bb2cd6c8f12b17b9ff53ead0b377ee88a"} Jul 29 14:01:18.935273 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.935123 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdzkw" event={"ID":"7e3ccd9f-be11-45b0-8380-fe38fdd54579","Type":"ContainerStarted","Data":"00deb0da11e2732369d982c9823fc63fd449e77218d141fb7593338c1a504f76"} Jul 29 14:01:18.956796 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.956699 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-st7fm" event={"ID":"33b01cb7-6fc6-4029-8054-bdd9a1043def","Type":"ContainerStarted","Data":"607a790ca0866fe3a7054ef534419d69a7c1ef33a7bd9da87e0cf332dae4dcd9"} Jul 29 14:01:18.963281 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.963228 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerStarted","Data":"59007461bb55025f19305a05922161aafef0ef9f3b6997613e3ebc023abe3a93"} Jul 29 14:01:18.975857 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.975812 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" event={"ID":"aa70b4877e9abd3c146a0f6d52419063","Type":"ContainerStarted","Data":"12f78c81bce402d347dcd0803d398f1d39f473371826b76efba2661afbf12d1b"} Jul 29 14:01:18.997063 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:18.996707 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8czqh" event={"ID":"af643c4d-5e6e-4ced-a948-653326504d7f","Type":"ContainerStarted","Data":"bc8dc8d96888558d397c46c469b28461b87408ab6a2d904d560670dd01652e2c"} Jul 29 14:01:19.004618 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:19.004516 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" event={"ID":"d78ddb82-d84a-4565-8bc9-018d47f5a549","Type":"ContainerStarted","Data":"eece9ce299c1d7228e59624b9a09faf4a3511f9f8030a87414781f01ad6cd946"} Jul 29 14:01:19.858936 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:19.858879 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:19.859523 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:19.859040 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:19.859615 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:19.859543 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:19.859690 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:19.859664 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:20.024450 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:20.024394 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"399ee5460b44b20743b00de1e70c4992","Type":"ContainerStarted","Data":"98d3d88260b779b48b9b844359ec670f2d15cda36bbf27f1ee1d0f0b93c90309"} Jul 29 14:01:20.041728 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:20.041655 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" podStartSLOduration=4.041623926 podStartE2EDuration="4.041623926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:18.992670481 +0000 UTC m=+4.645264319" watchObservedRunningTime="2026-07-29 14:01:20.041623926 +0000 UTC m=+5.694217765" Jul 29 14:01:20.373126 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:20.373083 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:20.373378 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:20.373340 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:20.373467 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:20.373391 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:20.373467 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:20.373407 2549 projected.go:196] Error preparing data for projected volume kube-api-access-7pwm5 for pod openshift-network-diagnostics/network-check-target-2cg6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:20.373588 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:20.373475 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5 podName:4cf66c81-3391-4691-8c9b-485548ca7061 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:24.373453125 +0000 UTC m=+10.026046954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pwm5" (UniqueName: "kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5") pod "network-check-target-2cg6g" (UID: "4cf66c81-3391-4691-8c9b-485548ca7061") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:20.473594 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:20.473549 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:20.473787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:20.473627 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:20.473787 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:20.473767 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:20.473939 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:20.473841 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs podName:ce6e0886-1615-4821-bba0-9cdba134b1d4 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:24.473819237 +0000 UTC m=+10.126413054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs") pod "network-metrics-daemon-nd944" (UID: "ce6e0886-1615-4821-bba0-9cdba134b1d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:20.474477 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:20.474319 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:20.474477 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:20.474385 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret podName:ba453020-a7c0-4204-988e-cfc912eafadb nodeName:}" failed. No retries permitted until 2026-07-29 14:01:24.474364028 +0000 UTC m=+10.126957846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret") pod "global-pull-secret-syncer-dwzkb" (UID: "ba453020-a7c0-4204-988e-cfc912eafadb") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:20.862634 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:20.861910 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:20.862634 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:20.862089 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:21.858514 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:21.858449 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:21.858756 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:21.858606 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:21.858833 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:21.858748 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:21.858954 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:21.858876 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:22.859864 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:22.859272 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:22.859864 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:22.859458 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:23.859105 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:23.859064 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:23.859105 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:23.859090 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:23.859435 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:23.859281 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:23.859602 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:23.859433 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:24.409217 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:24.409153 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:24.410098 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:24.409393 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:24.410098 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:24.409425 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:24.410098 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:24.409441 2549 projected.go:196] Error preparing data for projected volume kube-api-access-7pwm5 for pod openshift-network-diagnostics/network-check-target-2cg6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:24.410098 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:24.409550 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5 podName:4cf66c81-3391-4691-8c9b-485548ca7061 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:32.409527583 +0000 UTC m=+18.062121410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pwm5" (UniqueName: "kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5") pod "network-check-target-2cg6g" (UID: "4cf66c81-3391-4691-8c9b-485548ca7061") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:24.510300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:24.510253 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:24.510550 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:24.510429 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:24.510728 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:24.510606 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:24.510728 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:24.510678 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:24.510876 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:24.510684 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs podName:ce6e0886-1615-4821-bba0-9cdba134b1d4 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:32.510661028 +0000 UTC m=+18.163254849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs") pod "network-metrics-daemon-nd944" (UID: "ce6e0886-1615-4821-bba0-9cdba134b1d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:24.510876 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:24.510782 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret podName:ba453020-a7c0-4204-988e-cfc912eafadb nodeName:}" failed. No retries permitted until 2026-07-29 14:01:32.510762115 +0000 UTC m=+18.163355944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret") pod "global-pull-secret-syncer-dwzkb" (UID: "ba453020-a7c0-4204-988e-cfc912eafadb") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:24.868168 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:24.867922 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:24.868168 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:24.868099 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:25.859076 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:25.859040 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:25.859678 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:25.859195 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:25.859678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:25.859255 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:25.859678 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:25.859415 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:26.858829 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:26.858791 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:26.859000 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:26.858956 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:27.859595 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:27.858976 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:27.859595 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:27.859145 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:27.859595 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:27.859198 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:27.859595 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:27.859349 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:28.859180 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:28.859141 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:28.859427 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:28.859302 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:29.858355 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:29.858293 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:29.858355 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:29.858343 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:29.858918 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:29.858453 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:29.858918 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:29.858561 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:30.858957 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:30.858919 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:30.859470 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:30.859060 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:31.858235 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:31.858143 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:31.858441 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:31.858262 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:31.858441 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:31.858331 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:31.858535 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:31.858444 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:32.478892 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:32.478847 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:32.479347 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:32.479007 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:32.479347 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:32.479036 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:32.479347 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:32.479051 2549 projected.go:196] Error preparing data for projected volume kube-api-access-7pwm5 for pod openshift-network-diagnostics/network-check-target-2cg6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:32.479347 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:32.479143 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5 podName:4cf66c81-3391-4691-8c9b-485548ca7061 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:48.479104281 +0000 UTC m=+34.131698110 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pwm5" (UniqueName: "kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5") pod "network-check-target-2cg6g" (UID: "4cf66c81-3391-4691-8c9b-485548ca7061") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:32.579982 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:32.579938 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:32.580182 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:32.580025 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:32.580182 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:32.580107 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:32.580182 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:32.580144 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:32.580384 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:32.580189 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs podName:ce6e0886-1615-4821-bba0-9cdba134b1d4 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:48.580171638 +0000 UTC m=+34.232765450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs") pod "network-metrics-daemon-nd944" (UID: "ce6e0886-1615-4821-bba0-9cdba134b1d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:32.580384 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:32.580208 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret podName:ba453020-a7c0-4204-988e-cfc912eafadb nodeName:}" failed. No retries permitted until 2026-07-29 14:01:48.580198503 +0000 UTC m=+34.232792315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret") pod "global-pull-secret-syncer-dwzkb" (UID: "ba453020-a7c0-4204-988e-cfc912eafadb") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:32.859023 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:32.858988 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:32.859224 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:32.859129 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:33.858555 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:33.858510 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:33.859023 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:33.858526 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:33.859023 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:33.858667 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:33.859023 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:33.858753 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:34.859077 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:34.859036 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:34.859546 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:34.859151 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:35.858803 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:35.858764 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:35.859010 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:35.858764 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:35.859010 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:35.858883 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:35.859010 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:35.858992 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:36.859061 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:36.859027 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:36.860201 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:36.859152 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:37.859210 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:37.858851 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:37.859949 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:37.858851 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:37.859949 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:37.859340 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:37.859949 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:37.859378 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:38.066040 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.066012 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:01:38.066476 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.066448 2549 generic.go:356] "Generic (PLEG): container finished" podID="83f582eb-331b-4728-9e63-a5c94499d8b3" containerID="a514ff722cb31924dd7a00aed1427dc4485a8b30e1cb1a5f6327d6ee253d77b5" exitCode=1 Jul 29 14:01:38.066581 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.066523 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerStarted","Data":"8c0113102036eeb61b052b351becd6f699f0f4e9d1231eaec4bcaad3932a7ac9"} Jul 29 14:01:38.066642 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.066616 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerStarted","Data":"3f71d3176f36d9bb966834baf20077e50aae7fe796391269641458355c9eca3b"} Jul 29 14:01:38.066695 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.066670 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerStarted","Data":"8b8be8d355016be778d99e58c34485b78f61e89228926fd29212d0071df51cde"} Jul 29 14:01:38.066750 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.066721 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerStarted","Data":"fc356bb70dd0b9a70e2f35117931ad2b0e7a835eca60e212875fdcceb88b177d"} Jul 29 14:01:38.066812 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.066770 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerDied","Data":"a514ff722cb31924dd7a00aed1427dc4485a8b30e1cb1a5f6327d6ee253d77b5"} Jul 29 14:01:38.066865 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.066822 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerStarted","Data":"d716b990a72286b40d677da54a6ce91f60d20d434c2a1aeb0269c8b02b43b3b6"} Jul 29 14:01:38.068015 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.067989 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8czqh" event={"ID":"af643c4d-5e6e-4ced-a948-653326504d7f","Type":"ContainerStarted","Data":"0c219bf86114e26b7e8a44078f2c5bbb382632b6dbac8db9e4ca3b54e99d5836"} Jul 29 14:01:38.069430 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.069402 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" event={"ID":"d78ddb82-d84a-4565-8bc9-018d47f5a549","Type":"ContainerStarted","Data":"bc34fca1208b7c50ed6eeb7d28c756f2e9ad5313579ebf01c91c6be867f367ac"} Jul 29 14:01:38.070695 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.070672 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8dml7" event={"ID":"06d8cf0b-e7fd-4c62-a798-c6e26bda4526","Type":"ContainerStarted","Data":"82543c5eca2daf7f51e17f6f4a3814878b78a716cf570ee6f42c2e9bebcaaaa2"} Jul 29 14:01:38.072024 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.071998 2549 generic.go:356] "Generic (PLEG): container finished" podID="618f9616-1888-442e-879b-1739233920c5" containerID="07fae69a21baca4002a1d9c757157c43ad357c84990627c8fbb1e26be799ac8e" exitCode=0 Jul 29 14:01:38.072115 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.072089 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerDied","Data":"07fae69a21baca4002a1d9c757157c43ad357c84990627c8fbb1e26be799ac8e"} Jul 29 14:01:38.073738 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.073715 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-twwwt" event={"ID":"07dd2825-35bc-4921-9b52-456e3a5df0f6","Type":"ContainerStarted","Data":"3aac0ef3cf922c7658bdf9290d5b5a98d9d693f9506bc213fdd261f3d145a4c8"} Jul 29 14:01:38.075356 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.075334 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vfwlj" event={"ID":"4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436","Type":"ContainerStarted","Data":"62873b2bc499628ef798c69d0f20eb44ea97507fa0aa3af56bef41ca6d67b898"} Jul 29 14:01:38.080355 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.080322 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdzkw" event={"ID":"7e3ccd9f-be11-45b0-8380-fe38fdd54579","Type":"ContainerStarted","Data":"ce8ea28eaa4a03e394aacc717a29e44c4ecc56371087f451c8eff28bda6d0607"} Jul 29 14:01:38.086475 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.086412 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" podStartSLOduration=22.086397311 podStartE2EDuration="22.086397311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:20.042283314 +0000 UTC m=+5.694877153" watchObservedRunningTime="2026-07-29 14:01:38.086397311 +0000 UTC m=+23.738991145" Jul 29 14:01:38.104851 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.104786 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-twwwt" podStartSLOduration=4.032107399 podStartE2EDuration="23.104769742s" totalImagesPullingTime="19.072662343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:38.104619057 +0000 UTC m=+23.757212891" watchObservedRunningTime="2026-07-29 14:01:38.104769742 +0000 UTC m=+23.757363575" Jul 29 14:01:38.105342 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.105295 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8czqh" podStartSLOduration=3.970179515 podStartE2EDuration="23.105286397s" totalImagesPullingTime="19.135106882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:38.08578808 +0000 UTC m=+23.738381915" watchObservedRunningTime="2026-07-29 14:01:38.105286397 +0000 UTC m=+23.757880231" Jul 29 14:01:38.121088 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.121029 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8dml7" podStartSLOduration=4.057978924 podStartE2EDuration="23.121012897s" totalImagesPullingTime="19.063033973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:38.120535102 +0000 UTC m=+23.773128937" watchObservedRunningTime="2026-07-29 14:01:38.121012897 +0000 UTC m=+23.773606733" Jul 29 14:01:38.161212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.161108 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wdzkw" podStartSLOduration=4.074273715 podStartE2EDuration="23.16109358s" totalImagesPullingTime="19.086819865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:38.160954919 +0000 UTC m=+23.813548755" watchObservedRunningTime="2026-07-29 14:01:38.16109358 +0000 UTC m=+23.813687413" Jul 29 14:01:38.176515 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.176461 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vfwlj" podStartSLOduration=4.09090611 podStartE2EDuration="23.176442878s" totalImagesPullingTime="19.085536768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:38.176339156 +0000 UTC m=+23.828932991" watchObservedRunningTime="2026-07-29 14:01:38.176442878 +0000 UTC m=+23.829036711" Jul 29 14:01:38.608228 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.608200 2549 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 14:01:38.807072 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.806858 2549 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T14:01:38.608224247Z","UUID":"14cb88af-74bf-423d-a9cd-1ff33eb3fe8a","Handler":null,"Name":"","Endpoint":""} Jul 29 14:01:38.808979 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.808949 2549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 14:01:38.809134 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.808995 2549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 14:01:38.858734 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:38.858700 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:38.858954 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:38.858848 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:39.054407 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:39.054369 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:39.055218 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:39.055183 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:39.085059 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:39.084950 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-st7fm" event={"ID":"33b01cb7-6fc6-4029-8054-bdd9a1043def","Type":"ContainerStarted","Data":"105325b454082985469ef8cf8255dd742c02649afa62dd44cbe263713e43456a"} Jul 29 14:01:39.087711 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:39.087635 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" event={"ID":"d78ddb82-d84a-4565-8bc9-018d47f5a549","Type":"ContainerStarted","Data":"f0ce9e551b2f1c33805efa3adface422551377e730253c7aa8b58cd543af32c1"} Jul 29 14:01:39.088490 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:39.088462 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:39.089046 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:39.089012 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8dml7" Jul 29 14:01:39.100190 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:39.100140 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-st7fm" podStartSLOduration=5.008584792 podStartE2EDuration="24.100122548s" totalImagesPullingTime="19.091537756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:39.099696977 +0000 UTC m=+24.752290812" watchObservedRunningTime="2026-07-29 14:01:39.100122548 +0000 UTC m=+24.752716407" Jul 29 14:01:39.858487 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:39.858454 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:39.858708 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:39.858452 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:39.858708 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:39.858572 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:39.858708 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:39.858649 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:40.095034 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:40.095007 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:01:40.098195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:40.098149 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" event={"ID":"d78ddb82-d84a-4565-8bc9-018d47f5a549","Type":"ContainerStarted","Data":"eff1f2b0625b008643ea8848aa6e6bc41d8bc6239514ce549d691a375c294b90"} Jul 29 14:01:40.118500 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:40.118370 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dh4s" podStartSLOduration=4.136953617 podStartE2EDuration="25.118345826s" totalImagesPullingTime="20.981392209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:40.117902198 +0000 UTC m=+25.770496053" watchObservedRunningTime="2026-07-29 14:01:40.118345826 +0000 UTC m=+25.770939663" Jul 29 14:01:40.858326 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:40.858284 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:40.858528 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:40.858468 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:41.104631 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:41.104439 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:01:41.105114 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:41.105075 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerStarted","Data":"65c7f3f44e154ed5a95fcd31ec9405af735ea8980a877101ed502c5f07f7d384"} Jul 29 14:01:41.859204 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:41.859166 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:41.859486 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:41.859166 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:41.859486 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:41.859342 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:41.859486 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:41.859433 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:42.109343 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:42.109252 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerStarted","Data":"1628408e77099804d34452c5a1cd7fccae39cfd8f127a253d6990d9cd5b14560"} Jul 29 14:01:42.859226 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:42.858948 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:42.859394 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:42.859233 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:43.113571 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.113463 2549 generic.go:356] "Generic (PLEG): container finished" podID="618f9616-1888-442e-879b-1739233920c5" containerID="1628408e77099804d34452c5a1cd7fccae39cfd8f127a253d6990d9cd5b14560" exitCode=0 Jul 29 14:01:43.113571 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.113551 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerDied","Data":"1628408e77099804d34452c5a1cd7fccae39cfd8f127a253d6990d9cd5b14560"} Jul 29 14:01:43.117270 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.117253 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:01:43.117646 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.117626 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerStarted","Data":"47e94ab59ea508230912ef8e2318ac7606f68ae1875f31e00d44cbcf32e5d6c4"} Jul 29 14:01:43.117880 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.117856 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:43.118007 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.117916 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:43.118007 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.117962 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:43.118221 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.118205 2549 scope.go:117] "RemoveContainer" containerID="a514ff722cb31924dd7a00aed1427dc4485a8b30e1cb1a5f6327d6ee253d77b5" Jul 29 14:01:43.136088 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.136058 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:43.139878 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.139857 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:01:43.859079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.859016 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:43.859079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:43.859062 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:43.859386 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:43.859138 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:43.859386 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:43.859255 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:44.124475 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.124449 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:01:44.124883 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.124844 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" event={"ID":"83f582eb-331b-4728-9e63-a5c94499d8b3","Type":"ContainerStarted","Data":"dd418563205f80d420770b314648375d65d14ef847daa666646cb598a6863d7c"} Jul 29 14:01:44.127248 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.127214 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerStarted","Data":"25ce93680fbbf0afef3ff87e9ac606140bf98a36021b1a7659fcf8751690eef7"} Jul 29 14:01:44.152112 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.152058 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" podStartSLOduration=9.984275666 podStartE2EDuration="29.152039621s" totalImagesPullingTime="19.167763955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:44.150572044 +0000 UTC m=+29.803165877" watchObservedRunningTime="2026-07-29 14:01:44.152039621 +0000 UTC m=+29.804633454" Jul 29 14:01:44.160046 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.160011 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dwzkb"] Jul 29 14:01:44.160213 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.160139 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:44.160262 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:44.160245 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:44.166847 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.166807 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nd944"] Jul 29 14:01:44.167054 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.166946 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:44.167116 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:44.167057 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:44.167515 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.167489 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2cg6g"] Jul 29 14:01:44.167623 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:44.167582 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:44.167660 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:44.167649 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:45.131663 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:45.131372 2549 generic.go:356] "Generic (PLEG): container finished" podID="618f9616-1888-442e-879b-1739233920c5" containerID="25ce93680fbbf0afef3ff87e9ac606140bf98a36021b1a7659fcf8751690eef7" exitCode=0 Jul 29 14:01:45.132049 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:45.131464 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerDied","Data":"25ce93680fbbf0afef3ff87e9ac606140bf98a36021b1a7659fcf8751690eef7"} Jul 29 14:01:45.858579 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:45.858530 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:45.858579 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:45.858577 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:45.858781 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:45.858632 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:45.858781 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:45.858647 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:45.858781 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:45.858715 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:45.858781 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:45.858770 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:46.140384 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:46.140281 2549 generic.go:356] "Generic (PLEG): container finished" podID="618f9616-1888-442e-879b-1739233920c5" containerID="3859a9a1205d5b60a6d7b61ffe8a1f26955507004d9186b8139932062184de76" exitCode=0 Jul 29 14:01:46.140384 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:46.140347 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerDied","Data":"3859a9a1205d5b60a6d7b61ffe8a1f26955507004d9186b8139932062184de76"} Jul 29 14:01:47.859211 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:47.859171 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:47.859998 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:47.859171 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:47.859998 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:47.859297 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:47.859998 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:47.859171 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:47.859998 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:47.859405 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:47.859998 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:47.859493 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:48.498832 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:48.498790 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:48.499051 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:48.498990 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:48.499051 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:48.499017 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:48.499051 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:48.499030 2549 projected.go:196] Error preparing data for projected volume kube-api-access-7pwm5 for pod openshift-network-diagnostics/network-check-target-2cg6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:48.499207 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:48.499102 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5 podName:4cf66c81-3391-4691-8c9b-485548ca7061 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:20.499078333 +0000 UTC m=+66.151672145 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pwm5" (UniqueName: "kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5") pod "network-check-target-2cg6g" (UID: "4cf66c81-3391-4691-8c9b-485548ca7061") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:48.599300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:48.599250 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:48.599512 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:48.599361 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:48.599512 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:48.599468 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:48.599512 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:48.599503 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:48.599654 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:48.599562 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs podName:ce6e0886-1615-4821-bba0-9cdba134b1d4 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:20.599538091 +0000 UTC m=+66.252131905 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs") pod "network-metrics-daemon-nd944" (UID: "ce6e0886-1615-4821-bba0-9cdba134b1d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:48.599654 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:48.599584 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret podName:ba453020-a7c0-4204-988e-cfc912eafadb nodeName:}" failed. No retries permitted until 2026-07-29 14:02:20.599573287 +0000 UTC m=+66.252167106 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret") pod "global-pull-secret-syncer-dwzkb" (UID: "ba453020-a7c0-4204-988e-cfc912eafadb") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:49.858474 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:49.858437 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:49.858474 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:49.858480 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:49.858975 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:49.858512 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:49.858975 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:49.858629 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2cg6g" podUID="4cf66c81-3391-4691-8c9b-485548ca7061" Jul 29 14:01:49.858975 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:49.858743 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nd944" podUID="ce6e0886-1615-4821-bba0-9cdba134b1d4" Jul 29 14:01:49.858975 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:49.858803 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dwzkb" podUID="ba453020-a7c0-4204-988e-cfc912eafadb" Jul 29 14:01:50.177527 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.177443 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeReady" Jul 29 14:01:50.177714 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.177621 2549 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 14:01:50.231886 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.231843 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kvbf4"] Jul 29 14:01:50.255713 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.255678 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6499h"] Jul 29 14:01:50.255894 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.255837 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.258916 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.258891 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 14:01:50.259071 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.259003 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v4vw8\"" Jul 29 14:01:50.259138 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.258888 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 14:01:50.281922 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.281891 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kvbf4"] Jul 29 14:01:50.282112 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.281943 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6499h"] Jul 29 14:01:50.282112 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.282043 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6499h" Jul 29 14:01:50.284865 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.284829 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vwvk9\"" Jul 29 14:01:50.285017 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.284961 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 14:01:50.285017 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.284994 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 14:01:50.285563 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.285542 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 14:01:50.413022 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.412987 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-config-volume\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.413022 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.413027 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89xks\" (UniqueName: \"kubernetes.io/projected/5f409ba5-0292-42bd-8997-7437b42b5a36-kube-api-access-89xks\") pod \"ingress-canary-6499h\" (UID: \"5f409ba5-0292-42bd-8997-7437b42b5a36\") " pod="openshift-ingress-canary/ingress-canary-6499h" Jul 29 14:01:50.413257 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.413080 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-tmp-dir\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.413257 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.413105 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.413257 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.413156 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbwh\" (UniqueName: \"kubernetes.io/projected/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-kube-api-access-tmbwh\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.413257 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.413187 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f409ba5-0292-42bd-8997-7437b42b5a36-cert\") pod \"ingress-canary-6499h\" (UID: \"5f409ba5-0292-42bd-8997-7437b42b5a36\") " pod="openshift-ingress-canary/ingress-canary-6499h" Jul 29 14:01:50.514024 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.513971 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-config-volume\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.514024 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.514029 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89xks\" (UniqueName: \"kubernetes.io/projected/5f409ba5-0292-42bd-8997-7437b42b5a36-kube-api-access-89xks\") pod \"ingress-canary-6499h\" (UID: \"5f409ba5-0292-42bd-8997-7437b42b5a36\") " pod="openshift-ingress-canary/ingress-canary-6499h" Jul 29 14:01:50.514330 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.514067 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-tmp-dir\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.514330 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.514102 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.514330 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.514164 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbwh\" (UniqueName: \"kubernetes.io/projected/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-kube-api-access-tmbwh\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.514330 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.514210 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f409ba5-0292-42bd-8997-7437b42b5a36-cert\") pod \"ingress-canary-6499h\" (UID: \"5f409ba5-0292-42bd-8997-7437b42b5a36\") " pod="openshift-ingress-canary/ingress-canary-6499h" Jul 29 14:01:50.514330 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:50.514274 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:50.514533 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:50.514371 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls podName:afb32e32-1d93-4bf0-8134-7df7b7e3bfb6 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:51.014347246 +0000 UTC m=+36.666941070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls") pod "dns-default-kvbf4" (UID: "afb32e32-1d93-4bf0-8134-7df7b7e3bfb6") : secret "dns-default-metrics-tls" not found Jul 29 14:01:50.514815 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.514759 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-config-volume\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.514935 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.514887 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-tmp-dir\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.519653 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.519627 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f409ba5-0292-42bd-8997-7437b42b5a36-cert\") pod \"ingress-canary-6499h\" (UID: \"5f409ba5-0292-42bd-8997-7437b42b5a36\") " pod="openshift-ingress-canary/ingress-canary-6499h" Jul 29 14:01:50.523259 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.523230 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89xks\" (UniqueName: \"kubernetes.io/projected/5f409ba5-0292-42bd-8997-7437b42b5a36-kube-api-access-89xks\") pod \"ingress-canary-6499h\" (UID: \"5f409ba5-0292-42bd-8997-7437b42b5a36\") " pod="openshift-ingress-canary/ingress-canary-6499h" Jul 29 14:01:50.523514 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.523490 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbwh\" (UniqueName: \"kubernetes.io/projected/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-kube-api-access-tmbwh\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:50.593603 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:50.593568 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6499h" Jul 29 14:01:51.019023 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.018973 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:51.019662 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:51.019147 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:51.019662 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:51.019219 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls podName:afb32e32-1d93-4bf0-8134-7df7b7e3bfb6 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:52.019196627 +0000 UTC m=+37.671790443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls") pod "dns-default-kvbf4" (UID: "afb32e32-1d93-4bf0-8134-7df7b7e3bfb6") : secret "dns-default-metrics-tls" not found Jul 29 14:01:51.858588 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.858547 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:01:51.858792 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.858593 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:01:51.858792 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.858593 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:01:51.862555 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.862513 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6wqn8\"" Jul 29 14:01:51.862555 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.862522 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 14:01:51.862814 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.862513 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 14:01:51.862814 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.862519 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 14:01:51.862814 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.862606 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7g4zm\"" Jul 29 14:01:51.862814 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:51.862522 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 14:01:52.028003 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:52.027742 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:52.028553 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:52.027925 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:52.028553 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:52.028175 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls podName:afb32e32-1d93-4bf0-8134-7df7b7e3bfb6 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:54.028151891 +0000 UTC m=+39.680745723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls") pod "dns-default-kvbf4" (UID: "afb32e32-1d93-4bf0-8134-7df7b7e3bfb6") : secret "dns-default-metrics-tls" not found Jul 29 14:01:52.095556 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:52.095520 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6499h"] Jul 29 14:01:52.099738 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:52.099697 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f409ba5_0292_42bd_8997_7437b42b5a36.slice/crio-a30ffe203122fe5ec87b94865feab6503ba104a12e5d80c2b621eafb125d3521 WatchSource:0}: Error finding container a30ffe203122fe5ec87b94865feab6503ba104a12e5d80c2b621eafb125d3521: Status 404 returned error can't find the container with id a30ffe203122fe5ec87b94865feab6503ba104a12e5d80c2b621eafb125d3521 Jul 29 14:01:52.157511 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:52.157419 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6499h" event={"ID":"5f409ba5-0292-42bd-8997-7437b42b5a36","Type":"ContainerStarted","Data":"a30ffe203122fe5ec87b94865feab6503ba104a12e5d80c2b621eafb125d3521"} Jul 29 14:01:53.164547 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:53.164507 2549 generic.go:356] "Generic (PLEG): container finished" podID="618f9616-1888-442e-879b-1739233920c5" containerID="793b0d239d7deb1314090f7a057a5127f8a36bb07d60a70acfa37d2690cf6171" exitCode=0 Jul 29 14:01:53.165117 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:53.164595 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerDied","Data":"793b0d239d7deb1314090f7a057a5127f8a36bb07d60a70acfa37d2690cf6171"} Jul 29 14:01:54.044683 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.044434 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:54.044877 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:54.044613 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:54.044877 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:54.044804 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls podName:afb32e32-1d93-4bf0-8134-7df7b7e3bfb6 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:58.044780266 +0000 UTC m=+43.697374080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls") pod "dns-default-kvbf4" (UID: "afb32e32-1d93-4bf0-8134-7df7b7e3bfb6") : secret "dns-default-metrics-tls" not found Jul 29 14:01:54.170551 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.170515 2549 generic.go:356] "Generic (PLEG): container finished" podID="618f9616-1888-442e-879b-1739233920c5" containerID="5a9fb4b4270130ad681121d8f6cf7725cf45697873fd146092621457dcc5f095" exitCode=0 Jul 29 14:01:54.171019 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.170566 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerDied","Data":"5a9fb4b4270130ad681121d8f6cf7725cf45697873fd146092621457dcc5f095"} Jul 29 14:01:54.721387 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.721349 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2hgjg"] Jul 29 14:01:54.740025 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.739994 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.742983 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.742877 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 14:01:54.742983 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.742901 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 14:01:54.743241 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.743178 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2hgjg"] Jul 29 14:01:54.743349 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.743305 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jrmfz\"" Jul 29 14:01:54.743523 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.743400 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 14:01:54.744179 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.744160 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 14:01:54.851176 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.851140 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.851176 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.851182 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-data-volume\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.851417 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.851201 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.851417 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.851268 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-crio-socket\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.851417 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.851288 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrsj\" (UniqueName: \"kubernetes.io/projected/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-api-access-mmrsj\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.951884 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.951858 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-crio-socket\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.952001 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.951892 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrsj\" (UniqueName: \"kubernetes.io/projected/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-api-access-mmrsj\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.952001 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.951945 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.952001 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.951964 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-data-volume\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.952001 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.951985 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.952150 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:54.952066 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 14:01:54.952150 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:54.952079 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:01:54.952150 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.952098 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-crio-socket\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.952150 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:54.952126 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls podName:5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef nodeName:}" failed. No retries permitted until 2026-07-29 14:01:55.452111449 +0000 UTC m=+41.104705260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hgjg" (UID: "5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef") : secret "insights-runtime-extractor-tls" not found Jul 29 14:01:54.952278 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:54.952155 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm podName:5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef nodeName:}" failed. No retries permitted until 2026-07-29 14:01:55.452133834 +0000 UTC m=+41.104727652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2hgjg" (UID: "5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef") : configmap "kube-rbac-proxy" not found Jul 29 14:01:54.952337 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.952297 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-data-volume\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:54.967000 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:54.966971 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrsj\" (UniqueName: \"kubernetes.io/projected/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-api-access-mmrsj\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:55.176816 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:55.176770 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chbrj" event={"ID":"618f9616-1888-442e-879b-1739233920c5","Type":"ContainerStarted","Data":"858f78481f9a23dbdc8dfc2a5ccaa873f848247679cfa828752344a4be599e6c"} Jul 29 14:01:55.178301 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:55.178276 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6499h" event={"ID":"5f409ba5-0292-42bd-8997-7437b42b5a36","Type":"ContainerStarted","Data":"2c9d5157eb087c6de4ef272d3a0a338968e86abc7001fe65868cf81210d65239"} Jul 29 14:01:55.200166 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:55.200036 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-chbrj" podStartSLOduration=9.562208113 podStartE2EDuration="40.200018401s" totalImagesPullingTime="30.637810288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:55.199802912 +0000 UTC m=+40.852396747" watchObservedRunningTime="2026-07-29 14:01:55.200018401 +0000 UTC m=+40.852612234" Jul 29 14:01:55.214853 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:55.214776 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6499h" podStartSLOduration=2.373591176 podStartE2EDuration="5.214755952s" totalImagesPullingTime="2.841164776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:55.214368218 +0000 UTC m=+40.866962052" watchObservedRunningTime="2026-07-29 14:01:55.214755952 +0000 UTC m=+40.867349786" Jul 29 14:01:55.456591 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:55.456494 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:55.456591 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:55.456547 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:55.456785 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:55.456614 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 14:01:55.456785 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:55.456693 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm podName:5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef nodeName:}" failed. No retries permitted until 2026-07-29 14:01:56.456672328 +0000 UTC m=+42.109266150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2hgjg" (UID: "5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef") : configmap "kube-rbac-proxy" not found Jul 29 14:01:55.456785 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:55.456705 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:01:55.456785 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:55.456755 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls podName:5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef nodeName:}" failed. No retries permitted until 2026-07-29 14:01:56.456743035 +0000 UTC m=+42.109336853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hgjg" (UID: "5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef") : secret "insights-runtime-extractor-tls" not found Jul 29 14:01:56.464070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.464021 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:56.464070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.464072 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:56.464575 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:56.464148 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 14:01:56.464575 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:56.464228 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm podName:5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef nodeName:}" failed. No retries permitted until 2026-07-29 14:01:58.464210755 +0000 UTC m=+44.116804570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2hgjg" (UID: "5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef") : configmap "kube-rbac-proxy" not found Jul 29 14:01:56.464575 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:56.464288 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:01:56.464575 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:56.464369 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls podName:5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef nodeName:}" failed. No retries permitted until 2026-07-29 14:01:58.464355264 +0000 UTC m=+44.116949076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hgjg" (UID: "5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef") : secret "insights-runtime-extractor-tls" not found Jul 29 14:01:56.478744 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.478707 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2dm5g"] Jul 29 14:01:56.496716 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.496684 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2dm5g"] Jul 29 14:01:56.496875 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.496807 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.499564 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.499540 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 29 14:01:56.499564 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.499544 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 29 14:01:56.500609 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.500591 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 29 14:01:56.500722 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.500637 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wcntm\"" Jul 29 14:01:56.500722 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.500678 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 29 14:01:56.665813 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.665768 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqh8\" (UniqueName: \"kubernetes.io/projected/e1ce0761-acbd-4aa2-8164-f2dd83a28386-kube-api-access-rmqh8\") pod \"service-ca-658d7b7c78-2dm5g\" (UID: \"e1ce0761-acbd-4aa2-8164-f2dd83a28386\") " pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.665813 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.665816 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e1ce0761-acbd-4aa2-8164-f2dd83a28386-signing-cabundle\") pod \"service-ca-658d7b7c78-2dm5g\" (UID: \"e1ce0761-acbd-4aa2-8164-f2dd83a28386\") " pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.666046 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.665860 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e1ce0761-acbd-4aa2-8164-f2dd83a28386-signing-key\") pod \"service-ca-658d7b7c78-2dm5g\" (UID: \"e1ce0761-acbd-4aa2-8164-f2dd83a28386\") " pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.766747 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.766715 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqh8\" (UniqueName: \"kubernetes.io/projected/e1ce0761-acbd-4aa2-8164-f2dd83a28386-kube-api-access-rmqh8\") pod \"service-ca-658d7b7c78-2dm5g\" (UID: \"e1ce0761-acbd-4aa2-8164-f2dd83a28386\") " pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.766867 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.766763 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e1ce0761-acbd-4aa2-8164-f2dd83a28386-signing-cabundle\") pod \"service-ca-658d7b7c78-2dm5g\" (UID: \"e1ce0761-acbd-4aa2-8164-f2dd83a28386\") " pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.766867 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.766803 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e1ce0761-acbd-4aa2-8164-f2dd83a28386-signing-key\") pod \"service-ca-658d7b7c78-2dm5g\" (UID: \"e1ce0761-acbd-4aa2-8164-f2dd83a28386\") " pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.767472 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.767443 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e1ce0761-acbd-4aa2-8164-f2dd83a28386-signing-cabundle\") pod \"service-ca-658d7b7c78-2dm5g\" (UID: \"e1ce0761-acbd-4aa2-8164-f2dd83a28386\") " pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.769335 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.769303 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e1ce0761-acbd-4aa2-8164-f2dd83a28386-signing-key\") pod \"service-ca-658d7b7c78-2dm5g\" (UID: \"e1ce0761-acbd-4aa2-8164-f2dd83a28386\") " pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.775738 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.775713 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqh8\" (UniqueName: \"kubernetes.io/projected/e1ce0761-acbd-4aa2-8164-f2dd83a28386-kube-api-access-rmqh8\") pod \"service-ca-658d7b7c78-2dm5g\" (UID: \"e1ce0761-acbd-4aa2-8164-f2dd83a28386\") " pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.805824 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.805780 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" Jul 29 14:01:56.896445 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.896418 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdzkw_7e3ccd9f-be11-45b0-8380-fe38fdd54579/dns-node-resolver/0.log" Jul 29 14:01:56.951979 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:56.951945 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2dm5g"] Jul 29 14:01:56.954898 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:01:56.954865 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ce0761_acbd_4aa2_8164_f2dd83a28386.slice/crio-647830d4d74b22fce997f7b50b663d977828ee5a503cc3408826369dafcc0b53 WatchSource:0}: Error finding container 647830d4d74b22fce997f7b50b663d977828ee5a503cc3408826369dafcc0b53: Status 404 returned error can't find the container with id 647830d4d74b22fce997f7b50b663d977828ee5a503cc3408826369dafcc0b53 Jul 29 14:01:57.184473 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:57.184375 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" event={"ID":"e1ce0761-acbd-4aa2-8164-f2dd83a28386","Type":"ContainerStarted","Data":"647830d4d74b22fce997f7b50b663d977828ee5a503cc3408826369dafcc0b53"} Jul 29 14:01:57.897956 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:57.897922 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vfwlj_4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436/node-ca/0.log" Jul 29 14:01:58.075981 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:58.075942 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:01:58.076157 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:58.076139 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:58.076240 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:58.076225 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls podName:afb32e32-1d93-4bf0-8134-7df7b7e3bfb6 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:06.076202887 +0000 UTC m=+51.728796711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls") pod "dns-default-kvbf4" (UID: "afb32e32-1d93-4bf0-8134-7df7b7e3bfb6") : secret "dns-default-metrics-tls" not found Jul 29 14:01:58.297358 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:58.297294 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6499h_5f409ba5-0292-42bd-8997-7437b42b5a36/serve-healthcheck-canary/0.log" Jul 29 14:01:58.479937 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:58.479890 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:58.479937 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:01:58.479939 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:01:58.480152 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:58.480016 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 14:01:58.480152 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:58.480044 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:01:58.480152 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:58.480090 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls podName:5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef nodeName:}" failed. No retries permitted until 2026-07-29 14:02:02.480074559 +0000 UTC m=+48.132668371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hgjg" (UID: "5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef") : secret "insights-runtime-extractor-tls" not found Jul 29 14:01:58.480152 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:01:58.480102 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm podName:5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef nodeName:}" failed. No retries permitted until 2026-07-29 14:02:02.480095927 +0000 UTC m=+48.132689739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2hgjg" (UID: "5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef") : configmap "kube-rbac-proxy" not found Jul 29 14:02:00.194006 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:00.193962 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" event={"ID":"e1ce0761-acbd-4aa2-8164-f2dd83a28386","Type":"ContainerStarted","Data":"0bab80653502bad74c7785effb05d956bb6e26c4995f20af15a44ea0bba2fe82"} Jul 29 14:02:00.212352 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:00.212274 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-2dm5g" podStartSLOduration=1.7004373560000001 podStartE2EDuration="4.212251764s" totalImagesPullingTime="2.511814408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:00.211781126 +0000 UTC m=+45.864374960" watchObservedRunningTime="2026-07-29 14:02:00.212251764 +0000 UTC m=+45.864845597" Jul 29 14:02:02.513064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:02.513004 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:02:02.513598 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:02.513082 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:02:02.513598 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:02:02.513264 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:02:02.513598 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:02:02.513378 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls podName:5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef nodeName:}" failed. No retries permitted until 2026-07-29 14:02:10.513351601 +0000 UTC m=+56.165945433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hgjg" (UID: "5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef") : secret "insights-runtime-extractor-tls" not found Jul 29 14:02:02.513598 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:02.513554 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:02:06.140757 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:06.140715 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:02:06.143257 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:06.143231 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb32e32-1d93-4bf0-8134-7df7b7e3bfb6-metrics-tls\") pod \"dns-default-kvbf4\" (UID: \"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6\") " pod="openshift-dns/dns-default-kvbf4" Jul 29 14:02:06.168531 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:06.168493 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kvbf4" Jul 29 14:02:06.341539 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:06.341503 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kvbf4"] Jul 29 14:02:06.344471 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:06.344443 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb32e32_1d93_4bf0_8134_7df7b7e3bfb6.slice/crio-b1e91404a432fc78b8df5399a63e5ccfdf06944907cc7243a3e990c9d349f980 WatchSource:0}: Error finding container b1e91404a432fc78b8df5399a63e5ccfdf06944907cc7243a3e990c9d349f980: Status 404 returned error can't find the container with id b1e91404a432fc78b8df5399a63e5ccfdf06944907cc7243a3e990c9d349f980 Jul 29 14:02:07.216182 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:07.216140 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kvbf4" event={"ID":"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6","Type":"ContainerStarted","Data":"b1e91404a432fc78b8df5399a63e5ccfdf06944907cc7243a3e990c9d349f980"} Jul 29 14:02:09.222888 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:09.222787 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kvbf4" event={"ID":"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6","Type":"ContainerStarted","Data":"0a546240a9c24bb2de7e462e0ca281e9e4764702a677d190794f598978787993"} Jul 29 14:02:09.222888 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:09.222843 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kvbf4" event={"ID":"afb32e32-1d93-4bf0-8134-7df7b7e3bfb6","Type":"ContainerStarted","Data":"162a2c3eedb5b9be8f6b96252363fbfa2a792f0f8ebfcc64f5b2e36c06d0acfa"} Jul 29 14:02:09.223485 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:09.222927 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kvbf4" Jul 29 14:02:09.243025 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:09.242970 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kvbf4" podStartSLOduration=17.057359853 podStartE2EDuration="19.242951398s" totalImagesPullingTime="2.185591545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:09.241625654 +0000 UTC m=+54.894219489" watchObservedRunningTime="2026-07-29 14:02:09.242951398 +0000 UTC m=+54.895545231" Jul 29 14:02:10.581012 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:10.580961 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:02:10.583536 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:10.583501 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hgjg\" (UID: \"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef\") " pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:02:10.650502 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:10.650446 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2hgjg" Jul 29 14:02:10.782990 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:10.782946 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2hgjg"] Jul 29 14:02:10.786717 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:10.786674 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c056bdd_bf6c_43b2_b5e7_63a8fa3d2fef.slice/crio-8e6bda422f86a5ffb055099fad2a2db7fa33ef4999acc81a4eb06050ce32da1b WatchSource:0}: Error finding container 8e6bda422f86a5ffb055099fad2a2db7fa33ef4999acc81a4eb06050ce32da1b: Status 404 returned error can't find the container with id 8e6bda422f86a5ffb055099fad2a2db7fa33ef4999acc81a4eb06050ce32da1b Jul 29 14:02:11.233294 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:11.233246 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hgjg" event={"ID":"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef","Type":"ContainerStarted","Data":"8e6bda422f86a5ffb055099fad2a2db7fa33ef4999acc81a4eb06050ce32da1b"} Jul 29 14:02:15.144221 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:15.144186 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnvb6" Jul 29 14:02:17.255117 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:17.255065 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hgjg" event={"ID":"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef","Type":"ContainerStarted","Data":"4d3f06194f0ed84ef82cb9b9ca286cd33e44013c7d8ad23f7ba57b113d8fcf01"} Jul 29 14:02:19.228283 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:19.228250 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kvbf4" Jul 29 14:02:20.267117 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.267081 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hgjg" event={"ID":"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef","Type":"ContainerStarted","Data":"95db1c090d2b88a09cb7c93522d198f5cb9b26e479e41a29217fb1a7ea65c974"} Jul 29 14:02:20.417028 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.416650 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j"] Jul 29 14:02:20.419142 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.419109 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2"] Jul 29 14:02:20.419467 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.419446 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" Jul 29 14:02:20.421354 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.421330 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.422495 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.422474 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 29 14:02:20.422849 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.422830 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 14:02:20.423560 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.423501 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 14:02:20.423560 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.423528 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 14:02:20.423729 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.423564 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-84ctz\"" Jul 29 14:02:20.423809 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.423790 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 29 14:02:20.432595 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.432565 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j"] Jul 29 14:02:20.433644 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.433617 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2"] Jul 29 14:02:20.464615 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.464580 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c299d0c5-07d3-4073-ba86-fcb2db667f9c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66798f46cb-8kd6j\" (UID: \"c299d0c5-07d3-4073-ba86-fcb2db667f9c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" Jul 29 14:02:20.464800 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.464638 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a-tmp\") pod \"klusterlet-addon-workmgr-85cb6655c5-spvs2\" (UID: \"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.464800 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.464670 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hphq\" (UniqueName: \"kubernetes.io/projected/9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a-kube-api-access-8hphq\") pod \"klusterlet-addon-workmgr-85cb6655c5-spvs2\" (UID: \"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.464800 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.464709 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkph\" (UniqueName: \"kubernetes.io/projected/c299d0c5-07d3-4073-ba86-fcb2db667f9c-kube-api-access-drkph\") pod \"managed-serviceaccount-addon-agent-66798f46cb-8kd6j\" (UID: \"c299d0c5-07d3-4073-ba86-fcb2db667f9c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" Jul 29 14:02:20.464800 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.464732 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a-klusterlet-config\") pod \"klusterlet-addon-workmgr-85cb6655c5-spvs2\" (UID: \"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.517627 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.517590 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c7f585cd7-xf89c"] Jul 29 14:02:20.519605 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.519587 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.522173 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.522141 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fnsd9\"" Jul 29 14:02:20.522429 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.522409 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 14:02:20.522505 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.522459 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 14:02:20.522723 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.522706 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 14:02:20.529646 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.529623 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 14:02:20.537685 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.537655 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c7f585cd7-xf89c"] Jul 29 14:02:20.565335 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565252 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-trusted-ca\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.565542 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565352 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c299d0c5-07d3-4073-ba86-fcb2db667f9c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66798f46cb-8kd6j\" (UID: \"c299d0c5-07d3-4073-ba86-fcb2db667f9c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" Jul 29 14:02:20.565542 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565376 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-installation-pull-secrets\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.565542 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565401 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-registry-certificates\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.565542 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565419 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwcz\" (UniqueName: \"kubernetes.io/projected/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-kube-api-access-xhwcz\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.565542 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565445 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a-tmp\") pod \"klusterlet-addon-workmgr-85cb6655c5-spvs2\" (UID: \"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.565542 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565463 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-ca-trust-extracted\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.565542 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565487 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hphq\" (UniqueName: \"kubernetes.io/projected/9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a-kube-api-access-8hphq\") pod \"klusterlet-addon-workmgr-85cb6655c5-spvs2\" (UID: \"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.566098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565556 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:02:20.566098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565587 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-bound-sa-token\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.566098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565616 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkph\" (UniqueName: \"kubernetes.io/projected/c299d0c5-07d3-4073-ba86-fcb2db667f9c-kube-api-access-drkph\") pod \"managed-serviceaccount-addon-agent-66798f46cb-8kd6j\" (UID: \"c299d0c5-07d3-4073-ba86-fcb2db667f9c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" Jul 29 14:02:20.566098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565660 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a-klusterlet-config\") pod \"klusterlet-addon-workmgr-85cb6655c5-spvs2\" (UID: \"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.566098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565691 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-image-registry-private-configuration\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.566098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565729 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-registry-tls\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.566098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.565872 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a-tmp\") pod \"klusterlet-addon-workmgr-85cb6655c5-spvs2\" (UID: \"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.568232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.568206 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a-klusterlet-config\") pod \"klusterlet-addon-workmgr-85cb6655c5-spvs2\" (UID: \"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.568382 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.568261 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 14:02:20.568382 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.568358 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c299d0c5-07d3-4073-ba86-fcb2db667f9c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66798f46cb-8kd6j\" (UID: \"c299d0c5-07d3-4073-ba86-fcb2db667f9c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" Jul 29 14:02:20.578696 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.578670 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 14:02:20.580131 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.580108 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkph\" (UniqueName: \"kubernetes.io/projected/c299d0c5-07d3-4073-ba86-fcb2db667f9c-kube-api-access-drkph\") pod \"managed-serviceaccount-addon-agent-66798f46cb-8kd6j\" (UID: \"c299d0c5-07d3-4073-ba86-fcb2db667f9c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" Jul 29 14:02:20.580250 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.580196 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hphq\" (UniqueName: \"kubernetes.io/projected/9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a-kube-api-access-8hphq\") pod \"klusterlet-addon-workmgr-85cb6655c5-spvs2\" (UID: \"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.589211 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.589184 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwm5\" (UniqueName: \"kubernetes.io/projected/4cf66c81-3391-4691-8c9b-485548ca7061-kube-api-access-7pwm5\") pod \"network-check-target-2cg6g\" (UID: \"4cf66c81-3391-4691-8c9b-485548ca7061\") " pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:02:20.666131 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666090 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:02:20.666131 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666141 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-image-registry-private-configuration\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666173 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-registry-tls\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666220 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-trusted-ca\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666247 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-installation-pull-secrets\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666277 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-registry-certificates\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666304 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwcz\" (UniqueName: \"kubernetes.io/projected/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-kube-api-access-xhwcz\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666356 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-ca-trust-extracted\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666394 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:02:20.667013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.666420 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-bound-sa-token\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667446 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.667366 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-ca-trust-extracted\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667446 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.667396 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-trusted-ca\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.667532 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.667452 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-registry-certificates\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.668811 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.668786 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 14:02:20.669041 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.669022 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-image-registry-private-configuration\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.669089 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.669065 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-installation-pull-secrets\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.669089 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.669076 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 14:02:20.669183 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.669166 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-registry-tls\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.674870 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.674850 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6wqn8\"" Jul 29 14:02:20.678698 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.678667 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwcz\" (UniqueName: \"kubernetes.io/projected/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-kube-api-access-xhwcz\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.678805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.678750 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba453020-a7c0-4204-988e-cfc912eafadb-original-pull-secret\") pod \"global-pull-secret-syncer-dwzkb\" (UID: \"ba453020-a7c0-4204-988e-cfc912eafadb\") " pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:02:20.678805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.678750 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e0886-1615-4821-bba0-9cdba134b1d4-metrics-certs\") pod \"network-metrics-daemon-nd944\" (UID: \"ce6e0886-1615-4821-bba0-9cdba134b1d4\") " pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:02:20.678874 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.678853 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9-bound-sa-token\") pod \"image-registry-c7f585cd7-xf89c\" (UID: \"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9\") " pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.681740 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.681721 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:02:20.685326 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.685295 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dwzkb" Jul 29 14:02:20.737414 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.737380 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" Jul 29 14:02:20.744198 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.743989 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:20.829522 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.829142 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:20.833907 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.833417 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2cg6g"] Jul 29 14:02:20.870747 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.870704 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dwzkb"] Jul 29 14:02:20.880125 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:20.880078 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba453020_a7c0_4204_988e_cfc912eafadb.slice/crio-2e0fe690fc87552668198af76a97a6f7edf5ea55459176d7864eac95d316f24f WatchSource:0}: Error finding container 2e0fe690fc87552668198af76a97a6f7edf5ea55459176d7864eac95d316f24f: Status 404 returned error can't find the container with id 2e0fe690fc87552668198af76a97a6f7edf5ea55459176d7864eac95d316f24f Jul 29 14:02:20.926865 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.926288 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j"] Jul 29 14:02:20.932568 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:20.932520 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc299d0c5_07d3_4073_ba86_fcb2db667f9c.slice/crio-ab3d00fbda4abf025cf7dac4c83f84b8dbfb4e2f00468dfbd301d776e7a0821d WatchSource:0}: Error finding container ab3d00fbda4abf025cf7dac4c83f84b8dbfb4e2f00468dfbd301d776e7a0821d: Status 404 returned error can't find the container with id ab3d00fbda4abf025cf7dac4c83f84b8dbfb4e2f00468dfbd301d776e7a0821d Jul 29 14:02:20.944241 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.944206 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2"] Jul 29 14:02:20.948087 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:20.948056 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee32ad8_d5d4_44a9_b0b4_a7e9924c903a.slice/crio-8fd8b909d570d0aad118f63cad7d095d9cedc8337b8974fc78cdd2de9b530209 WatchSource:0}: Error finding container 8fd8b909d570d0aad118f63cad7d095d9cedc8337b8974fc78cdd2de9b530209: Status 404 returned error can't find the container with id 8fd8b909d570d0aad118f63cad7d095d9cedc8337b8974fc78cdd2de9b530209 Jul 29 14:02:20.982457 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.982421 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7g4zm\"" Jul 29 14:02:20.988740 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:20.988713 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nd944" Jul 29 14:02:21.011966 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.011932 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c7f585cd7-xf89c"] Jul 29 14:02:21.016389 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:21.016347 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17adb8d_ff91_4c2d_aa69_5e7bb71f2eb9.slice/crio-bcd2c5166521b2d34ff594c1d81db6baceb0f556bc52801aa28e28851e14b362 WatchSource:0}: Error finding container bcd2c5166521b2d34ff594c1d81db6baceb0f556bc52801aa28e28851e14b362: Status 404 returned error can't find the container with id bcd2c5166521b2d34ff594c1d81db6baceb0f556bc52801aa28e28851e14b362 Jul 29 14:02:21.144357 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.144276 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nd944"] Jul 29 14:02:21.148357 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:21.148277 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6e0886_1615_4821_bba0_9cdba134b1d4.slice/crio-42db186252273ded085fa34c47bccd6b1458a3bc4dbf5b86f39ab7e7f72913d6 WatchSource:0}: Error finding container 42db186252273ded085fa34c47bccd6b1458a3bc4dbf5b86f39ab7e7f72913d6: Status 404 returned error can't find the container with id 42db186252273ded085fa34c47bccd6b1458a3bc4dbf5b86f39ab7e7f72913d6 Jul 29 14:02:21.272941 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.272904 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" event={"ID":"c299d0c5-07d3-4073-ba86-fcb2db667f9c","Type":"ContainerStarted","Data":"ab3d00fbda4abf025cf7dac4c83f84b8dbfb4e2f00468dfbd301d776e7a0821d"} Jul 29 14:02:21.275057 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.275022 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dwzkb" event={"ID":"ba453020-a7c0-4204-988e-cfc912eafadb","Type":"ContainerStarted","Data":"2e0fe690fc87552668198af76a97a6f7edf5ea55459176d7864eac95d316f24f"} Jul 29 14:02:21.277043 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.277010 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" event={"ID":"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9","Type":"ContainerStarted","Data":"7b34b046c94f0a8aea57039df1f0e0c60103598dea65e5d2a8c78b48b081ea8a"} Jul 29 14:02:21.277190 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.277067 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" event={"ID":"d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9","Type":"ContainerStarted","Data":"bcd2c5166521b2d34ff594c1d81db6baceb0f556bc52801aa28e28851e14b362"} Jul 29 14:02:21.277190 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.277141 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:21.278707 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.278670 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" event={"ID":"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a","Type":"ContainerStarted","Data":"8fd8b909d570d0aad118f63cad7d095d9cedc8337b8974fc78cdd2de9b530209"} Jul 29 14:02:21.280424 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.280393 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2cg6g" event={"ID":"4cf66c81-3391-4691-8c9b-485548ca7061","Type":"ContainerStarted","Data":"f5f916f438b5fd05c17060567002fddbf7edc10f1f34b6f930588b9516e96269"} Jul 29 14:02:21.282567 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.282504 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nd944" event={"ID":"ce6e0886-1615-4821-bba0-9cdba134b1d4","Type":"ContainerStarted","Data":"42db186252273ded085fa34c47bccd6b1458a3bc4dbf5b86f39ab7e7f72913d6"} Jul 29 14:02:21.297784 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:21.297219 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" podStartSLOduration=1.29719394 podStartE2EDuration="1.29719394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:21.295632445 +0000 UTC m=+66.948226280" watchObservedRunningTime="2026-07-29 14:02:21.29719394 +0000 UTC m=+66.949787775" Jul 29 14:02:23.304275 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:23.303266 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nd944" event={"ID":"ce6e0886-1615-4821-bba0-9cdba134b1d4","Type":"ContainerStarted","Data":"19aec454de4a55139666059154cf63d24df04e73c4bb14b8b811c7e43d69fa72"} Jul 29 14:02:23.304275 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:23.303354 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nd944" event={"ID":"ce6e0886-1615-4821-bba0-9cdba134b1d4","Type":"ContainerStarted","Data":"687c0817a2299ee9b33d433755dadc6c004dcde917a0b5af899fb69bf370edc9"} Jul 29 14:02:23.322616 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:23.321261 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nd944" podStartSLOduration=66.69668623 podStartE2EDuration="1m8.321238385s" totalImagesPullingTime="1.624552155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:23.3208918 +0000 UTC m=+68.973485639" watchObservedRunningTime="2026-07-29 14:02:23.321238385 +0000 UTC m=+68.973832223" Jul 29 14:02:24.411556 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:24.411516 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx"] Jul 29 14:02:24.415591 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:24.415560 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" Jul 29 14:02:24.418201 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:24.418171 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 29 14:02:24.418502 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:24.418483 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-c8rbv\"" Jul 29 14:02:24.421362 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:24.421334 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx"] Jul 29 14:02:24.504756 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:24.504710 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/42bedec4-cbdd-49ff-934a-e6d0144d18ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-drvnx\" (UID: \"42bedec4-cbdd-49ff-934a-e6d0144d18ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" Jul 29 14:02:24.605387 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:24.605340 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/42bedec4-cbdd-49ff-934a-e6d0144d18ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-drvnx\" (UID: \"42bedec4-cbdd-49ff-934a-e6d0144d18ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" Jul 29 14:02:24.605592 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:02:24.605491 2549 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 29 14:02:24.605592 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:02:24.605580 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42bedec4-cbdd-49ff-934a-e6d0144d18ab-tls-certificates podName:42bedec4-cbdd-49ff-934a-e6d0144d18ab nodeName:}" failed. No retries permitted until 2026-07-29 14:02:25.105554628 +0000 UTC m=+70.758148443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/42bedec4-cbdd-49ff-934a-e6d0144d18ab-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-drvnx" (UID: "42bedec4-cbdd-49ff-934a-e6d0144d18ab") : secret "prometheus-operator-admission-webhook-tls" not found Jul 29 14:02:25.112638 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:25.112592 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/42bedec4-cbdd-49ff-934a-e6d0144d18ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-drvnx\" (UID: \"42bedec4-cbdd-49ff-934a-e6d0144d18ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" Jul 29 14:02:25.115154 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:25.115118 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/42bedec4-cbdd-49ff-934a-e6d0144d18ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-drvnx\" (UID: \"42bedec4-cbdd-49ff-934a-e6d0144d18ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" Jul 29 14:02:25.331816 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:25.331771 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" Jul 29 14:02:31.160724 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.160600 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx"] Jul 29 14:02:31.163852 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:31.163806 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bedec4_cbdd_49ff_934a_e6d0144d18ab.slice/crio-84e3f43d568b5acce5c717a02456654720c5cc442f1a14a94a52735293943643 WatchSource:0}: Error finding container 84e3f43d568b5acce5c717a02456654720c5cc442f1a14a94a52735293943643: Status 404 returned error can't find the container with id 84e3f43d568b5acce5c717a02456654720c5cc442f1a14a94a52735293943643 Jul 29 14:02:31.334925 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.334886 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2cg6g" event={"ID":"4cf66c81-3391-4691-8c9b-485548ca7061","Type":"ContainerStarted","Data":"bb539c73f3fdf4b61fbef8d2d9dff7d5a236eb3353bc1042e1573b86a928dd07"} Jul 29 14:02:31.335164 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.334972 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:02:31.336728 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.336696 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" event={"ID":"c299d0c5-07d3-4073-ba86-fcb2db667f9c","Type":"ContainerStarted","Data":"ee06e1185543fb8fb5f017b342154f30cb1e120aec43b5295af0ce5e76407857"} Jul 29 14:02:31.338205 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.338172 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dwzkb" event={"ID":"ba453020-a7c0-4204-988e-cfc912eafadb","Type":"ContainerStarted","Data":"9306229ea3367397dc17f1c8b8b6884a7aa388482dab338a24975835cbb37fd8"} Jul 29 14:02:31.339519 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.339482 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" event={"ID":"42bedec4-cbdd-49ff-934a-e6d0144d18ab","Type":"ContainerStarted","Data":"84e3f43d568b5acce5c717a02456654720c5cc442f1a14a94a52735293943643"} Jul 29 14:02:31.341681 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.341656 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hgjg" event={"ID":"5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef","Type":"ContainerStarted","Data":"6db43663c7db82dfc46f827534c6057686d3448b9cbb6814425d8e554b99f283"} Jul 29 14:02:31.343140 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.343117 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" event={"ID":"9ee32ad8-d5d4-44a9-b0b4-a7e9924c903a","Type":"ContainerStarted","Data":"30e62d447c8a950538e86d50cc49dd8bb69e1e0a4de2f2d5f2c91f6f093460cd"} Jul 29 14:02:31.343345 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.343328 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:31.345192 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.345171 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" Jul 29 14:02:31.350574 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.350529 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2cg6g" podStartSLOduration=66.197870165 podStartE2EDuration="1m16.350513204s" totalImagesPullingTime="10.152643039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:31.349105612 +0000 UTC m=+77.001699446" watchObservedRunningTime="2026-07-29 14:02:31.350513204 +0000 UTC m=+77.003107037" Jul 29 14:02:31.363442 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.363378 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66798f46cb-8kd6j" podStartSLOduration=1.301325367 podStartE2EDuration="11.363360253s" totalImagesPullingTime="10.062034886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:31.362800806 +0000 UTC m=+77.015394640" watchObservedRunningTime="2026-07-29 14:02:31.363360253 +0000 UTC m=+77.015954087" Jul 29 14:02:31.376906 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.376848 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dwzkb" podStartSLOduration=65.250335155 podStartE2EDuration="1m15.376833088s" totalImagesPullingTime="10.126497933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:31.375970064 +0000 UTC m=+77.028563899" watchObservedRunningTime="2026-07-29 14:02:31.376833088 +0000 UTC m=+77.029426921" Jul 29 14:02:31.400978 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.400920 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2hgjg" podStartSLOduration=17.300563853 podStartE2EDuration="37.400903112s" totalImagesPullingTime="20.100339259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:31.399767841 +0000 UTC m=+77.052361717" watchObservedRunningTime="2026-07-29 14:02:31.400903112 +0000 UTC m=+77.053496945" Jul 29 14:02:31.416787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:31.416732 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85cb6655c5-spvs2" podStartSLOduration=1.352949996 podStartE2EDuration="11.416713914s" totalImagesPullingTime="10.063763918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:31.415659639 +0000 UTC m=+77.068253474" watchObservedRunningTime="2026-07-29 14:02:31.416713914 +0000 UTC m=+77.069307748" Jul 29 14:02:33.352787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:33.352745 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" event={"ID":"42bedec4-cbdd-49ff-934a-e6d0144d18ab","Type":"ContainerStarted","Data":"db0e41b65ec6b1470a4207ba15a5f932bc7899df7191634e91303b794fd2f9ab"} Jul 29 14:02:33.353278 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:33.353139 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" Jul 29 14:02:33.357796 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:33.357760 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" Jul 29 14:02:33.373669 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:33.373599 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-drvnx" podStartSLOduration=8.107063674 podStartE2EDuration="9.373579897s" totalImagesPullingTime="1.266516223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:33.373176051 +0000 UTC m=+79.025769885" watchObservedRunningTime="2026-07-29 14:02:33.373579897 +0000 UTC m=+79.026173733" Jul 29 14:02:38.820886 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.820738 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4"] Jul 29 14:02:38.823994 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.823970 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:38.827578 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.827550 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 29 14:02:38.827793 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.827778 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 29 14:02:38.828069 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.828044 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dmj9m\"" Jul 29 14:02:38.828364 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.828340 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 14:02:38.828722 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.828701 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 14:02:38.829045 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.829022 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 14:02:38.837683 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.837656 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4"] Jul 29 14:02:38.863375 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.863347 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ltcpg"] Jul 29 14:02:38.865910 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.865888 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:38.868571 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.868529 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 14:02:38.868803 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.868595 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vrdkg\"" Jul 29 14:02:38.868803 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.868596 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 14:02:38.868803 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.868787 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 14:02:38.877499 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.877471 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4"] Jul 29 14:02:38.879900 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.879878 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:38.883555 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.883488 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 29 14:02:38.883820 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.883802 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mkblc\"" Jul 29 14:02:38.883928 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.883911 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 29 14:02:38.884044 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.884030 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 29 14:02:38.907816 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.907779 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4"] Jul 29 14:02:38.921138 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.921106 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0827515-4442-4ff1-b78a-03647cf0afcc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:38.921336 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.921162 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqhw\" (UniqueName: \"kubernetes.io/projected/a0827515-4442-4ff1-b78a-03647cf0afcc-kube-api-access-lqqhw\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:38.921336 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.921231 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0827515-4442-4ff1-b78a-03647cf0afcc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:38.921336 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:38.921274 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0827515-4442-4ff1-b78a-03647cf0afcc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.022531 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022490 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/010ceb81-e2a2-4531-94c2-1f8b03bdabec-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.022694 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022547 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.022694 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022571 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.022694 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022593 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.022694 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022623 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/992ee83e-02f2-4a61-b574-89b3e33177ef-sys\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.022694 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022680 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-wtmp\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.022879 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022702 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/010ceb81-e2a2-4531-94c2-1f8b03bdabec-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.022879 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022756 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqhw\" (UniqueName: \"kubernetes.io/projected/a0827515-4442-4ff1-b78a-03647cf0afcc-kube-api-access-lqqhw\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.022879 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022789 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/992ee83e-02f2-4a61-b574-89b3e33177ef-root\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.022879 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022812 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-textfile\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.022879 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022831 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992ee83e-02f2-4a61-b574-89b3e33177ef-metrics-client-ca\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.023037 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022884 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.023037 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022908 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0827515-4442-4ff1-b78a-03647cf0afcc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.023037 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022946 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0827515-4442-4ff1-b78a-03647cf0afcc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.023037 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.022969 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-tls\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.023037 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.023010 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0827515-4442-4ff1-b78a-03647cf0afcc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.023231 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.023038 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.023231 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.023066 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqhc4\" (UniqueName: \"kubernetes.io/projected/992ee83e-02f2-4a61-b574-89b3e33177ef-kube-api-access-kqhc4\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.023231 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.023096 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvj6v\" (UniqueName: \"kubernetes.io/projected/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-api-access-dvj6v\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.023630 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.023607 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0827515-4442-4ff1-b78a-03647cf0afcc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.025552 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.025528 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0827515-4442-4ff1-b78a-03647cf0afcc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.025836 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.025816 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0827515-4442-4ff1-b78a-03647cf0afcc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.032631 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.032604 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqhw\" (UniqueName: \"kubernetes.io/projected/a0827515-4442-4ff1-b78a-03647cf0afcc-kube-api-access-lqqhw\") pod \"openshift-state-metrics-b8f49f5c6-nx4x4\" (UID: \"a0827515-4442-4ff1-b78a-03647cf0afcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.123700 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.123586 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.123700 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.123645 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqhc4\" (UniqueName: \"kubernetes.io/projected/992ee83e-02f2-4a61-b574-89b3e33177ef-kube-api-access-kqhc4\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.123957 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.123812 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvj6v\" (UniqueName: \"kubernetes.io/projected/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-api-access-dvj6v\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.123957 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.123853 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/010ceb81-e2a2-4531-94c2-1f8b03bdabec-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.124074 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.123960 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.124074 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.123989 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.124074 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124021 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.124074 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124051 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/992ee83e-02f2-4a61-b574-89b3e33177ef-sys\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124283 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124101 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-wtmp\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124283 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124126 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/010ceb81-e2a2-4531-94c2-1f8b03bdabec-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.124283 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124162 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/992ee83e-02f2-4a61-b574-89b3e33177ef-root\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124283 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124195 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/992ee83e-02f2-4a61-b574-89b3e33177ef-sys\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124503 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124350 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-wtmp\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124503 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124401 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/992ee83e-02f2-4a61-b574-89b3e33177ef-root\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124503 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124196 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-textfile\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124503 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124450 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992ee83e-02f2-4a61-b574-89b3e33177ef-metrics-client-ca\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124503 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124486 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124743 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124519 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-tls\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124743 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124545 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-textfile\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.124743 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124679 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/010ceb81-e2a2-4531-94c2-1f8b03bdabec-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.124743 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124692 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/010ceb81-e2a2-4531-94c2-1f8b03bdabec-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.124743 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.124713 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.125035 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.125009 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.125176 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.125081 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992ee83e-02f2-4a61-b574-89b3e33177ef-metrics-client-ca\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.126298 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.126276 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.126877 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.126849 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.127057 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.127039 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.127168 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.127148 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/992ee83e-02f2-4a61-b574-89b3e33177ef-node-exporter-tls\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.131618 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.131594 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvj6v\" (UniqueName: \"kubernetes.io/projected/010ceb81-e2a2-4531-94c2-1f8b03bdabec-kube-api-access-dvj6v\") pod \"kube-state-metrics-56d9f944b4-p82s4\" (UID: \"010ceb81-e2a2-4531-94c2-1f8b03bdabec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.132201 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.132168 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqhc4\" (UniqueName: \"kubernetes.io/projected/992ee83e-02f2-4a61-b574-89b3e33177ef-kube-api-access-kqhc4\") pod \"node-exporter-ltcpg\" (UID: \"992ee83e-02f2-4a61-b574-89b3e33177ef\") " pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.134975 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.134951 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" Jul 29 14:02:39.175210 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.175170 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ltcpg" Jul 29 14:02:39.186957 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:39.186920 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992ee83e_02f2_4a61_b574_89b3e33177ef.slice/crio-5d02ceccee9c473461d8a8840f27f2901378963c91526e835a2fce84aec100f1 WatchSource:0}: Error finding container 5d02ceccee9c473461d8a8840f27f2901378963c91526e835a2fce84aec100f1: Status 404 returned error can't find the container with id 5d02ceccee9c473461d8a8840f27f2901378963c91526e835a2fce84aec100f1 Jul 29 14:02:39.188369 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.188347 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" Jul 29 14:02:39.291635 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.291501 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4"] Jul 29 14:02:39.374886 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.374734 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltcpg" event={"ID":"992ee83e-02f2-4a61-b574-89b3e33177ef","Type":"ContainerStarted","Data":"5d02ceccee9c473461d8a8840f27f2901378963c91526e835a2fce84aec100f1"} Jul 29 14:02:39.375672 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.375440 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4"] Jul 29 14:02:39.376357 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:39.376333 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" event={"ID":"a0827515-4442-4ff1-b78a-03647cf0afcc","Type":"ContainerStarted","Data":"23ad709d2334f78909b1f9fb2694de7f18f25e8cc40720b09ac9d4d7a97db475"} Jul 29 14:02:39.378507 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:39.378483 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010ceb81_e2a2_4531_94c2_1f8b03bdabec.slice/crio-fa06a01969ef7a22b402da22e7fce432024b99fa66ec6044aa37a06d80453bd0 WatchSource:0}: Error finding container fa06a01969ef7a22b402da22e7fce432024b99fa66ec6044aa37a06d80453bd0: Status 404 returned error can't find the container with id fa06a01969ef7a22b402da22e7fce432024b99fa66ec6044aa37a06d80453bd0 Jul 29 14:02:40.381846 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:40.381799 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" event={"ID":"010ceb81-e2a2-4531-94c2-1f8b03bdabec","Type":"ContainerStarted","Data":"fa06a01969ef7a22b402da22e7fce432024b99fa66ec6044aa37a06d80453bd0"} Jul 29 14:02:40.384059 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:40.384024 2549 generic.go:356] "Generic (PLEG): container finished" podID="992ee83e-02f2-4a61-b574-89b3e33177ef" containerID="5f566747be5abfdbb21a86e3ee1022a3f0c7934949582c0c83b99b5d56d6504c" exitCode=0 Jul 29 14:02:40.384212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:40.384116 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltcpg" event={"ID":"992ee83e-02f2-4a61-b574-89b3e33177ef","Type":"ContainerDied","Data":"5f566747be5abfdbb21a86e3ee1022a3f0c7934949582c0c83b99b5d56d6504c"} Jul 29 14:02:40.386371 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:40.386341 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" event={"ID":"a0827515-4442-4ff1-b78a-03647cf0afcc","Type":"ContainerStarted","Data":"0f7bc6e572bb76e1fb5c1c704722aae01c9816dcf1eed7483459a491aa16d766"} Jul 29 14:02:40.386472 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:40.386405 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" event={"ID":"a0827515-4442-4ff1-b78a-03647cf0afcc","Type":"ContainerStarted","Data":"965519a7b105960c4f3fa43ff1e9112f53aa59a281216a085946a4fdaa1a84f2"} Jul 29 14:02:41.394387 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:41.394353 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltcpg" event={"ID":"992ee83e-02f2-4a61-b574-89b3e33177ef","Type":"ContainerStarted","Data":"3fc8f31283c01ef42fa460f2a9d651eb556e7c1f7f4691f85faaa3eec98031a4"} Jul 29 14:02:41.394811 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:41.394424 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltcpg" event={"ID":"992ee83e-02f2-4a61-b574-89b3e33177ef","Type":"ContainerStarted","Data":"a2560498e9a26bf36e183595e2243c6460cbd6899f59862dda99c7920f53e332"} Jul 29 14:02:41.397117 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:41.397083 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" event={"ID":"010ceb81-e2a2-4531-94c2-1f8b03bdabec","Type":"ContainerStarted","Data":"bc0dd0dc4543bb75a37420e3aeaa401351136bc0e24a1930f8215232710ba97a"} Jul 29 14:02:41.397219 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:41.397151 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" event={"ID":"010ceb81-e2a2-4531-94c2-1f8b03bdabec","Type":"ContainerStarted","Data":"ccab77d934eae5bb91f271dba61efe87a1e71bc6ae0874c30b6e2568cc6b9ae2"} Jul 29 14:02:41.422659 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:41.422610 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ltcpg" podStartSLOduration=2.38341787 podStartE2EDuration="3.422595557s" totalImagesPullingTime="958.736484ms" totalInitContainerRuntime="80.441203ms" isStatefulPod=true podCreationTimestamp="2026-07-29 14:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:41.422408128 +0000 UTC m=+87.075001963" watchObservedRunningTime="2026-07-29 14:02:41.422595557 +0000 UTC m=+87.075189390" Jul 29 14:02:42.293376 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.293342 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c7f585cd7-xf89c" Jul 29 14:02:42.305161 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.305133 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-5wwjd"] Jul 29 14:02:42.307547 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.307529 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-5wwjd" Jul 29 14:02:42.310057 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.310034 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bngxm\"" Jul 29 14:02:42.310205 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.310036 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 14:02:42.310205 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.310036 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 14:02:42.316103 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.316071 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-5wwjd"] Jul 29 14:02:42.405959 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.405859 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" event={"ID":"a0827515-4442-4ff1-b78a-03647cf0afcc","Type":"ContainerStarted","Data":"d238029dcd633234bd91f7688d59388ddbe29485abc1413c655ad972dcc6b656"} Jul 29 14:02:42.408016 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.407983 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" event={"ID":"010ceb81-e2a2-4531-94c2-1f8b03bdabec","Type":"ContainerStarted","Data":"126bb082d26be158c630953e98d98b5da192e01c616aab896eae047ecbafa8b2"} Jul 29 14:02:42.427427 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.427366 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nx4x4" podStartSLOduration=1.766399407 podStartE2EDuration="4.427349726s" totalImagesPullingTime="2.660950319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:42.42725813 +0000 UTC m=+88.079851964" watchObservedRunningTime="2026-07-29 14:02:42.427349726 +0000 UTC m=+88.079943560" Jul 29 14:02:42.445457 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.445399 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-p82s4" podStartSLOduration=2.7304703359999998 podStartE2EDuration="4.445377109s" totalImagesPullingTime="1.714906773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:42.444041803 +0000 UTC m=+88.096635638" watchObservedRunningTime="2026-07-29 14:02:42.445377109 +0000 UTC m=+88.097970943" Jul 29 14:02:42.453392 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.453350 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrm7v\" (UniqueName: \"kubernetes.io/projected/83ad3cff-e751-4953-b740-70df9ce69f39-kube-api-access-vrm7v\") pod \"downloads-7c97dc4555-5wwjd\" (UID: \"83ad3cff-e751-4953-b740-70df9ce69f39\") " pod="openshift-console/downloads-7c97dc4555-5wwjd" Jul 29 14:02:42.554719 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.554651 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrm7v\" (UniqueName: \"kubernetes.io/projected/83ad3cff-e751-4953-b740-70df9ce69f39-kube-api-access-vrm7v\") pod \"downloads-7c97dc4555-5wwjd\" (UID: \"83ad3cff-e751-4953-b740-70df9ce69f39\") " pod="openshift-console/downloads-7c97dc4555-5wwjd" Jul 29 14:02:42.563696 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.563656 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrm7v\" (UniqueName: \"kubernetes.io/projected/83ad3cff-e751-4953-b740-70df9ce69f39-kube-api-access-vrm7v\") pod \"downloads-7c97dc4555-5wwjd\" (UID: \"83ad3cff-e751-4953-b740-70df9ce69f39\") " pod="openshift-console/downloads-7c97dc4555-5wwjd" Jul 29 14:02:42.617637 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.617590 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-5wwjd" Jul 29 14:02:42.758083 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:42.758055 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-5wwjd"] Jul 29 14:02:42.760727 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:42.760691 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ad3cff_e751_4953_b740_70df9ce69f39.slice/crio-092f5897a55062d1b273e837a2ec22ac521cf1bba798d946e51d18838fcbb032 WatchSource:0}: Error finding container 092f5897a55062d1b273e837a2ec22ac521cf1bba798d946e51d18838fcbb032: Status 404 returned error can't find the container with id 092f5897a55062d1b273e837a2ec22ac521cf1bba798d946e51d18838fcbb032 Jul 29 14:02:43.262934 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.262894 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-848c946485-6r9bg"] Jul 29 14:02:43.266333 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.266292 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.269886 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.269857 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 29 14:02:43.270118 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.269856 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2qwjv\"" Jul 29 14:02:43.270118 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.269952 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 29 14:02:43.270118 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.269912 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a27timgkt7i55\"" Jul 29 14:02:43.270118 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.269911 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 29 14:02:43.270118 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.269858 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 14:02:43.276220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.276189 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-848c946485-6r9bg"] Jul 29 14:02:43.363121 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.363074 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-secret-metrics-server-client-certs\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.363121 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.363119 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-secret-metrics-server-tls\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.363384 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.363138 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-audit-log\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.363384 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.363238 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.363384 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.363299 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4z5l\" (UniqueName: \"kubernetes.io/projected/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-kube-api-access-p4z5l\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.363384 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.363361 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-metrics-server-audit-profiles\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.363513 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.363384 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-client-ca-bundle\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.412865 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.412827 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-5wwjd" event={"ID":"83ad3cff-e751-4953-b740-70df9ce69f39","Type":"ContainerStarted","Data":"092f5897a55062d1b273e837a2ec22ac521cf1bba798d946e51d18838fcbb032"} Jul 29 14:02:43.464792 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.464749 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4z5l\" (UniqueName: \"kubernetes.io/projected/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-kube-api-access-p4z5l\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.464792 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.464796 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-metrics-server-audit-profiles\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.465064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.464820 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-client-ca-bundle\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.465064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.464869 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-secret-metrics-server-client-certs\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.465064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.464902 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-secret-metrics-server-tls\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.465064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.464929 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-audit-log\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.465064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.464980 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.465399 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.465374 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-audit-log\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.465720 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.465694 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.466069 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.466045 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-metrics-server-audit-profiles\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.467744 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.467711 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-secret-metrics-server-client-certs\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.468000 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.467981 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-client-ca-bundle\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.468141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.468122 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-secret-metrics-server-tls\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.473178 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.473152 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4z5l\" (UniqueName: \"kubernetes.io/projected/a5b43b8f-e98f-4c6a-9e9c-f83145239ba5-kube-api-access-p4z5l\") pod \"metrics-server-848c946485-6r9bg\" (UID: \"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5\") " pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.577501 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.577395 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:02:43.731055 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:43.731017 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-848c946485-6r9bg"] Jul 29 14:02:43.734727 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:43.734698 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b43b8f_e98f_4c6a_9e9c_f83145239ba5.slice/crio-f8e95d0341525759cc9b4243303d54ee42d825fa9ea78b6715ec0acf410a6b4e WatchSource:0}: Error finding container f8e95d0341525759cc9b4243303d54ee42d825fa9ea78b6715ec0acf410a6b4e: Status 404 returned error can't find the container with id f8e95d0341525759cc9b4243303d54ee42d825fa9ea78b6715ec0acf410a6b4e Jul 29 14:02:44.070984 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.070947 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f88b49d44-hjs9z"] Jul 29 14:02:44.073494 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.073469 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.076165 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.076105 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4p6bm\"" Jul 29 14:02:44.076165 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.076131 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 29 14:02:44.076414 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.076166 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 29 14:02:44.076414 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.076184 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 29 14:02:44.076414 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.076115 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 29 14:02:44.076575 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.076516 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 29 14:02:44.081629 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.081604 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 29 14:02:44.089424 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.089391 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f88b49d44-hjs9z"] Jul 29 14:02:44.169448 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.169412 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-federate-client-tls\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.169667 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.169466 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e491a-10ea-4c8a-b1d2-a1b41b910e90-serving-certs-ca-bundle\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.169667 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.169521 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zh5\" (UniqueName: \"kubernetes.io/projected/634e491a-10ea-4c8a-b1d2-a1b41b910e90-kube-api-access-84zh5\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.169667 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.169557 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-telemeter-client-tls\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.169667 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.169582 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-secret-telemeter-client\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.169667 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.169601 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634e491a-10ea-4c8a-b1d2-a1b41b910e90-metrics-client-ca\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.169667 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.169622 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.170048 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.169726 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e491a-10ea-4c8a-b1d2-a1b41b910e90-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.270704 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.270667 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-federate-client-tls\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.270988 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.270719 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e491a-10ea-4c8a-b1d2-a1b41b910e90-serving-certs-ca-bundle\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.270988 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.270742 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zh5\" (UniqueName: \"kubernetes.io/projected/634e491a-10ea-4c8a-b1d2-a1b41b910e90-kube-api-access-84zh5\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.270988 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.270811 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-telemeter-client-tls\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.270988 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.270861 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-secret-telemeter-client\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.270988 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.270892 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634e491a-10ea-4c8a-b1d2-a1b41b910e90-metrics-client-ca\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.270988 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.270921 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.270988 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.270982 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e491a-10ea-4c8a-b1d2-a1b41b910e90-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.272450 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.272419 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634e491a-10ea-4c8a-b1d2-a1b41b910e90-metrics-client-ca\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.272626 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.272504 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e491a-10ea-4c8a-b1d2-a1b41b910e90-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.272626 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.272527 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e491a-10ea-4c8a-b1d2-a1b41b910e90-serving-certs-ca-bundle\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.284713 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.284631 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-telemeter-client-tls\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.288007 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.287908 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-federate-client-tls\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.288590 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.288558 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-secret-telemeter-client\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.289455 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.289420 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/634e491a-10ea-4c8a-b1d2-a1b41b910e90-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.291745 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.291713 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zh5\" (UniqueName: \"kubernetes.io/projected/634e491a-10ea-4c8a-b1d2-a1b41b910e90-kube-api-access-84zh5\") pod \"telemeter-client-5f88b49d44-hjs9z\" (UID: \"634e491a-10ea-4c8a-b1d2-a1b41b910e90\") " pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.386085 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.385985 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" Jul 29 14:02:44.422695 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.422575 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-848c946485-6r9bg" event={"ID":"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5","Type":"ContainerStarted","Data":"f8e95d0341525759cc9b4243303d54ee42d825fa9ea78b6715ec0acf410a6b4e"} Jul 29 14:02:44.572704 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:44.572605 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f88b49d44-hjs9z"] Jul 29 14:02:44.576657 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:44.576621 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634e491a_10ea_4c8a_b1d2_a1b41b910e90.slice/crio-bb3780246917db386322987f855170d87ab294309b45b03cde56e5fa550c953a WatchSource:0}: Error finding container bb3780246917db386322987f855170d87ab294309b45b03cde56e5fa550c953a: Status 404 returned error can't find the container with id bb3780246917db386322987f855170d87ab294309b45b03cde56e5fa550c953a Jul 29 14:02:45.060011 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.059968 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 14:02:45.063800 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.063767 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.066787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.066742 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 29 14:02:45.067394 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.067365 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 29 14:02:45.067544 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.067370 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 29 14:02:45.067905 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.067883 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r6b8z\"" Jul 29 14:02:45.067993 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.067889 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aspjhen8bhjqh\"" Jul 29 14:02:45.067993 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.067940 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 29 14:02:45.069079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.068245 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 29 14:02:45.069079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.068269 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 29 14:02:45.069079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.068426 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 29 14:02:45.069079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.068495 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 29 14:02:45.069079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.068606 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 29 14:02:45.069079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.068654 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 29 14:02:45.069079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.068661 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 29 14:02:45.069079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.068970 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 29 14:02:45.069631 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.069609 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 29 14:02:45.071276 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.071077 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 29 14:02:45.081626 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.081546 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 14:02:45.179099 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179055 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-config\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179362 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179111 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-web-config\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179362 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179159 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179362 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179188 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179362 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179259 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-config-out\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179362 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179286 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179362 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179337 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179368 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179425 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgl7\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-kube-api-access-fqgl7\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179470 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179501 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179544 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179580 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179622 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179647 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179684 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.179712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179708 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.180130 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179739 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.180130 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179772 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.180130 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.179797 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280331 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280272 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280551 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280360 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgl7\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-kube-api-access-fqgl7\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280551 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280400 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280551 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280458 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280551 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280494 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280551 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280519 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280554 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280596 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280630 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280655 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280681 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280712 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280736 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280766 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-config\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.280817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280791 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-web-config\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.281237 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280869 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.281237 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280899 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.281237 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280928 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-config-out\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.281237 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.280971 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.281237 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.281014 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.281577 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.281370 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.281816 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.281784 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.283031 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.282709 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.283031 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.282899 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.284859 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.284782 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.285025 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.284998 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.285860 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.285831 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.286015 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.285845 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.286212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.286189 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.287111 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.286897 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-config\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.287111 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.287069 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.289067 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.289028 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.290266 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.290229 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.290382 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.290278 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.290591 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.290565 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-config-out\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.290745 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.290701 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-web-config\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.290854 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.290764 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.291020 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.290989 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.291994 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.291970 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.292929 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.292879 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgl7\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-kube-api-access-fqgl7\") pod \"prometheus-k8s-0\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.378523 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.378483 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:02:45.431069 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.431010 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" event={"ID":"634e491a-10ea-4c8a-b1d2-a1b41b910e90","Type":"ContainerStarted","Data":"bb3780246917db386322987f855170d87ab294309b45b03cde56e5fa550c953a"} Jul 29 14:02:45.435602 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.435546 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-848c946485-6r9bg" event={"ID":"a5b43b8f-e98f-4c6a-9e9c-f83145239ba5","Type":"ContainerStarted","Data":"95e71bc8d39a77f02a2e4a50f09b6ed0f96aaf91d628d561ceab89edfbf23e72"} Jul 29 14:02:45.576831 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.574894 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-848c946485-6r9bg" podStartSLOduration=0.991428079 podStartE2EDuration="2.574872097s" totalImagesPullingTime="1.583444018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:45.455505706 +0000 UTC m=+91.108099537" watchObservedRunningTime="2026-07-29 14:02:45.574872097 +0000 UTC m=+91.227465930" Jul 29 14:02:45.576831 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:45.576672 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 14:02:45.584395 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:02:45.584013 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7605399_8eb1_4a7e_a070_c8bb681c4445.slice/crio-a5d2e50dc161a76edbf337f57d11392de7644e0fc7b302665262a1076a7021a6 WatchSource:0}: Error finding container a5d2e50dc161a76edbf337f57d11392de7644e0fc7b302665262a1076a7021a6: Status 404 returned error can't find the container with id a5d2e50dc161a76edbf337f57d11392de7644e0fc7b302665262a1076a7021a6 Jul 29 14:02:46.443409 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:46.443271 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerStarted","Data":"a5d2e50dc161a76edbf337f57d11392de7644e0fc7b302665262a1076a7021a6"} Jul 29 14:02:47.451825 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:47.450962 2549 generic.go:356] "Generic (PLEG): container finished" podID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerID="6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d" exitCode=0 Jul 29 14:02:47.451825 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:47.451078 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerDied","Data":"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d"} Jul 29 14:02:49.468204 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:49.468156 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" event={"ID":"634e491a-10ea-4c8a-b1d2-a1b41b910e90","Type":"ContainerStarted","Data":"06152357628e69d02eff9e87f0e1fb1fcc09969703a621391e1af6118cff5731"} Jul 29 14:02:49.468661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:49.468237 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" event={"ID":"634e491a-10ea-4c8a-b1d2-a1b41b910e90","Type":"ContainerStarted","Data":"44efcea5368e075aa6a03a5a52e51a9cc1bbfe8e062add9a776faf4aa993c9c1"} Jul 29 14:02:49.468661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:49.468271 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" event={"ID":"634e491a-10ea-4c8a-b1d2-a1b41b910e90","Type":"ContainerStarted","Data":"92c7773761106c45fca47c08855830655f60c557405e694a7d11f6f712c69cbb"} Jul 29 14:02:49.495723 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:49.495624 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f88b49d44-hjs9z" podStartSLOduration=1.026313243 podStartE2EDuration="5.49559702s" totalImagesPullingTime="4.469283777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:49.494259321 +0000 UTC m=+95.146853193" watchObservedRunningTime="2026-07-29 14:02:49.49559702 +0000 UTC m=+95.148190855" Jul 29 14:02:51.484433 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:51.484388 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerStarted","Data":"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007"} Jul 29 14:02:52.491920 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:02:52.491873 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerStarted","Data":"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721"} Jul 29 14:03:02.350554 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:02.350520 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2cg6g" Jul 29 14:03:03.553209 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.553164 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerStarted","Data":"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b"} Jul 29 14:03:03.553786 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.553265 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerStarted","Data":"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5"} Jul 29 14:03:03.553786 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.553334 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerStarted","Data":"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4"} Jul 29 14:03:03.553786 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.553378 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerStarted","Data":"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5"} Jul 29 14:03:03.555062 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.555026 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-5wwjd" event={"ID":"83ad3cff-e751-4953-b740-70df9ce69f39","Type":"ContainerStarted","Data":"b253deb7622bbb75b660d6ea0ae80689d8cb5dc2514e7daedb29793539019dfa"} Jul 29 14:03:03.555202 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.555096 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-5wwjd" Jul 29 14:03:03.568662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.568629 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-5wwjd" Jul 29 14:03:03.577780 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.577735 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:03:03.577990 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.577830 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:03:03.583609 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.583546 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.404335791 podStartE2EDuration="18.58352078s" totalImagesPullingTime="16.154966812s" totalInitContainerRuntime="24.218177ms" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:03:03.580384303 +0000 UTC m=+109.232978140" watchObservedRunningTime="2026-07-29 14:03:03.58352078 +0000 UTC m=+109.236114615" Jul 29 14:03:03.598606 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:03.598546 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-5wwjd" podStartSLOduration=1.811944597 podStartE2EDuration="21.598525451s" totalImagesPullingTime="19.786580854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:03:03.596742427 +0000 UTC m=+109.249336284" watchObservedRunningTime="2026-07-29 14:03:03.598525451 +0000 UTC m=+109.251119286" Jul 29 14:03:05.379549 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:05.379503 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:03:12.130975 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:12.130940 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-p82s4_010ceb81-e2a2-4531-94c2-1f8b03bdabec/kube-state-metrics/0.log" Jul 29 14:03:12.330580 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:12.330541 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-p82s4_010ceb81-e2a2-4531-94c2-1f8b03bdabec/kube-rbac-proxy-main/0.log" Jul 29 14:03:12.530071 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:12.530041 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-p82s4_010ceb81-e2a2-4531-94c2-1f8b03bdabec/kube-rbac-proxy-self/0.log" Jul 29 14:03:12.730615 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:12.730584 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-848c946485-6r9bg_a5b43b8f-e98f-4c6a-9e9c-f83145239ba5/metrics-server/0.log" Jul 29 14:03:14.330562 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:14.330526 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltcpg_992ee83e-02f2-4a61-b574-89b3e33177ef/init-textfile/0.log" Jul 29 14:03:14.530661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:14.530631 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltcpg_992ee83e-02f2-4a61-b574-89b3e33177ef/node-exporter/0.log" Jul 29 14:03:14.729850 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:14.729771 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltcpg_992ee83e-02f2-4a61-b574-89b3e33177ef/kube-rbac-proxy/0.log" Jul 29 14:03:14.931619 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:14.931575 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nx4x4_a0827515-4442-4ff1-b78a-03647cf0afcc/kube-rbac-proxy-main/0.log" Jul 29 14:03:15.130078 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:15.130048 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nx4x4_a0827515-4442-4ff1-b78a-03647cf0afcc/kube-rbac-proxy-self/0.log" Jul 29 14:03:15.330448 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:15.330418 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nx4x4_a0827515-4442-4ff1-b78a-03647cf0afcc/openshift-state-metrics/0.log" Jul 29 14:03:15.529969 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:15.529940 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a7605399-8eb1-4a7e-a070-c8bb681c4445/init-config-reloader/0.log" Jul 29 14:03:15.731842 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:15.731814 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a7605399-8eb1-4a7e-a070-c8bb681c4445/prometheus/0.log" Jul 29 14:03:15.930441 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:15.930367 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a7605399-8eb1-4a7e-a070-c8bb681c4445/config-reloader/0.log" Jul 29 14:03:16.130258 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:16.130226 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a7605399-8eb1-4a7e-a070-c8bb681c4445/thanos-sidecar/0.log" Jul 29 14:03:16.330335 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:16.330279 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a7605399-8eb1-4a7e-a070-c8bb681c4445/kube-rbac-proxy-web/0.log" Jul 29 14:03:16.530655 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:16.530627 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a7605399-8eb1-4a7e-a070-c8bb681c4445/kube-rbac-proxy/0.log" Jul 29 14:03:16.730787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:16.730705 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a7605399-8eb1-4a7e-a070-c8bb681c4445/kube-rbac-proxy-thanos/0.log" Jul 29 14:03:17.330855 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:17.330817 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-drvnx_42bedec4-cbdd-49ff-934a-e6d0144d18ab/prometheus-operator-admission-webhook/0.log" Jul 29 14:03:17.530515 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:17.530486 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f88b49d44-hjs9z_634e491a-10ea-4c8a-b1d2-a1b41b910e90/telemeter-client/0.log" Jul 29 14:03:17.730452 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:17.730361 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f88b49d44-hjs9z_634e491a-10ea-4c8a-b1d2-a1b41b910e90/reload/0.log" Jul 29 14:03:17.930428 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:17.930392 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f88b49d44-hjs9z_634e491a-10ea-4c8a-b1d2-a1b41b910e90/kube-rbac-proxy/0.log" Jul 29 14:03:20.330816 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:20.330783 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-5wwjd_83ad3cff-e751-4953-b740-70df9ce69f39/download-server/0.log" Jul 29 14:03:20.730433 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:20.730340 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6499h_5f409ba5-0292-42bd-8997-7437b42b5a36/serve-healthcheck-canary/0.log" Jul 29 14:03:23.590578 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:23.590540 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:03:23.597376 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:23.597344 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-848c946485-6r9bg" Jul 29 14:03:45.379141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:45.379092 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:03:45.443948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:45.443916 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:03:45.765034 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:03:45.765004 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:03.423363 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.423323 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 14:04:03.423947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.423646 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="prometheus" containerID="cri-o://528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" gracePeriod=600 Jul 29 14:04:03.423947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.423686 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" gracePeriod=600 Jul 29 14:04:03.423947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.423663 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy" containerID="cri-o://4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" gracePeriod=600 Jul 29 14:04:03.423947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.423725 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="thanos-sidecar" containerID="cri-o://5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" gracePeriod=600 Jul 29 14:04:03.423947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.423734 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="config-reloader" containerID="cri-o://f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" gracePeriod=600 Jul 29 14:04:03.423947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.423729 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy-web" containerID="cri-o://c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" gracePeriod=600 Jul 29 14:04:03.669876 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.669847 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:03.819181 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819146 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-kubelet-serving-ca-bundle\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819449 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819194 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-2\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819449 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819232 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-thanos-prometheus-http-client-file\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819449 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819258 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-db\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819449 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819284 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-1\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819449 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819307 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-trusted-ca-bundle\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819449 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819358 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqgl7\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-kube-api-access-fqgl7\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819449 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819383 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-0\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819449 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819430 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-kube-rbac-proxy\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819458 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-config\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819530 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-metrics-client-certs\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819562 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-web-config\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819587 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-tls\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819620 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-serving-certs-ca-bundle\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819653 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819679 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819727 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-config-out\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819731 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819764 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-metrics-client-ca\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.819805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819799 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-grpc-tls\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.820336 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.819834 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-tls-assets\") pod \"a7605399-8eb1-4a7e-a070-c8bb681c4445\" (UID: \"a7605399-8eb1-4a7e-a070-c8bb681c4445\") " Jul 29 14:04:03.820336 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.820122 2549 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.821219 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.820573 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:04:03.821219 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.820929 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:04:03.821219 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.820958 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:04:03.821219 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.821081 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:04:03.821526 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.821457 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:04:03.822787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.822407 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:03.822787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.822485 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:03.822787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.822493 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-kube-api-access-fqgl7" (OuterVolumeSpecName: "kube-api-access-fqgl7") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "kube-api-access-fqgl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:04:03.822787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.822704 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:04:03.823161 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.822899 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:04:03.823684 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.823652 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:03.823826 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.823798 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-config" (OuterVolumeSpecName: "config") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:03.824241 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.824209 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:04:03.824241 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.824225 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:03.825006 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.824984 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-config-out" (OuterVolumeSpecName: "config-out") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:04:03.825306 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.825286 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:03.825655 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.825634 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:03.826027 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.826010 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:03.827505 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827478 2549 generic.go:356] "Generic (PLEG): container finished" podID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerID="7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" exitCode=0 Jul 29 14:04:03.827505 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827501 2549 generic.go:356] "Generic (PLEG): container finished" podID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerID="4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" exitCode=0 Jul 29 14:04:03.827505 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827508 2549 generic.go:356] "Generic (PLEG): container finished" podID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerID="c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" exitCode=0 Jul 29 14:04:03.827653 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827514 2549 generic.go:356] "Generic (PLEG): container finished" podID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerID="5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" exitCode=0 Jul 29 14:04:03.827653 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827519 2549 generic.go:356] "Generic (PLEG): container finished" podID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerID="f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" exitCode=0 Jul 29 14:04:03.827653 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827525 2549 generic.go:356] "Generic (PLEG): container finished" podID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerID="528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" exitCode=0 Jul 29 14:04:03.827653 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827563 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerDied","Data":"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b"} Jul 29 14:04:03.827653 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827593 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:03.827815 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827663 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerDied","Data":"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5"} Jul 29 14:04:03.827815 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827695 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerDied","Data":"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4"} Jul 29 14:04:03.827815 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827729 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerDied","Data":"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5"} Jul 29 14:04:03.827815 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827767 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerDied","Data":"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721"} Jul 29 14:04:03.827815 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827791 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerDied","Data":"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007"} Jul 29 14:04:03.827815 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827731 2549 scope.go:117] "RemoveContainer" containerID="7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" Jul 29 14:04:03.828032 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.827827 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7605399-8eb1-4a7e-a070-c8bb681c4445","Type":"ContainerDied","Data":"a5d2e50dc161a76edbf337f57d11392de7644e0fc7b302665262a1076a7021a6"} Jul 29 14:04:03.837286 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.837268 2549 scope.go:117] "RemoveContainer" containerID="4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" Jul 29 14:04:03.838175 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.838148 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-web-config" (OuterVolumeSpecName: "web-config") pod "a7605399-8eb1-4a7e-a070-c8bb681c4445" (UID: "a7605399-8eb1-4a7e-a070-c8bb681c4445"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:03.844974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.844950 2549 scope.go:117] "RemoveContainer" containerID="c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" Jul 29 14:04:03.852249 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.852229 2549 scope.go:117] "RemoveContainer" containerID="5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" Jul 29 14:04:03.859689 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.859667 2549 scope.go:117] "RemoveContainer" containerID="f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" Jul 29 14:04:03.867173 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.867152 2549 scope.go:117] "RemoveContainer" containerID="528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" Jul 29 14:04:03.875054 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.875033 2549 scope.go:117] "RemoveContainer" containerID="6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d" Jul 29 14:04:03.883005 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.882982 2549 scope.go:117] "RemoveContainer" containerID="7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" Jul 29 14:04:03.883352 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:04:03.883327 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": container with ID starting with 7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b not found: ID does not exist" containerID="7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" Jul 29 14:04:03.883418 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.883365 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b"} err="failed to get container status \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": rpc error: code = NotFound desc = could not find container \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": container with ID starting with 7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b not found: ID does not exist" Jul 29 14:04:03.883418 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.883400 2549 scope.go:117] "RemoveContainer" containerID="4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" Jul 29 14:04:03.883688 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:04:03.883668 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": container with ID starting with 4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5 not found: ID does not exist" containerID="4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" Jul 29 14:04:03.883729 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.883696 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5"} err="failed to get container status \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": rpc error: code = NotFound desc = could not find container \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": container with ID starting with 4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5 not found: ID does not exist" Jul 29 14:04:03.883729 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.883712 2549 scope.go:117] "RemoveContainer" containerID="c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" Jul 29 14:04:03.883995 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:04:03.883977 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": container with ID starting with c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4 not found: ID does not exist" containerID="c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" Jul 29 14:04:03.884036 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.884002 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4"} err="failed to get container status \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": rpc error: code = NotFound desc = could not find container \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": container with ID starting with c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4 not found: ID does not exist" Jul 29 14:04:03.884036 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.884017 2549 scope.go:117] "RemoveContainer" containerID="5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" Jul 29 14:04:03.884257 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:04:03.884240 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": container with ID starting with 5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5 not found: ID does not exist" containerID="5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" Jul 29 14:04:03.884302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.884262 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5"} err="failed to get container status \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": rpc error: code = NotFound desc = could not find container \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": container with ID starting with 5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5 not found: ID does not exist" Jul 29 14:04:03.884302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.884278 2549 scope.go:117] "RemoveContainer" containerID="f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" Jul 29 14:04:03.884517 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:04:03.884501 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": container with ID starting with f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721 not found: ID does not exist" containerID="f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" Jul 29 14:04:03.884560 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.884519 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721"} err="failed to get container status \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": rpc error: code = NotFound desc = could not find container \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": container with ID starting with f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721 not found: ID does not exist" Jul 29 14:04:03.884560 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.884533 2549 scope.go:117] "RemoveContainer" containerID="528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" Jul 29 14:04:03.884750 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:04:03.884734 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": container with ID starting with 528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007 not found: ID does not exist" containerID="528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" Jul 29 14:04:03.884830 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.884754 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007"} err="failed to get container status \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": rpc error: code = NotFound desc = could not find container \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": container with ID starting with 528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007 not found: ID does not exist" Jul 29 14:04:03.884830 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.884769 2549 scope.go:117] "RemoveContainer" containerID="6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d" Jul 29 14:04:03.884983 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:04:03.884965 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": container with ID starting with 6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d not found: ID does not exist" containerID="6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d" Jul 29 14:04:03.885024 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.884990 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d"} err="failed to get container status \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": rpc error: code = NotFound desc = could not find container \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": container with ID starting with 6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d not found: ID does not exist" Jul 29 14:04:03.885024 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.885005 2549 scope.go:117] "RemoveContainer" containerID="7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" Jul 29 14:04:03.885237 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.885220 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b"} err="failed to get container status \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": rpc error: code = NotFound desc = could not find container \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": container with ID starting with 7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b not found: ID does not exist" Jul 29 14:04:03.885283 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.885237 2549 scope.go:117] "RemoveContainer" containerID="4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" Jul 29 14:04:03.885497 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.885474 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5"} err="failed to get container status \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": rpc error: code = NotFound desc = could not find container \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": container with ID starting with 4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5 not found: ID does not exist" Jul 29 14:04:03.885586 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.885504 2549 scope.go:117] "RemoveContainer" containerID="c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" Jul 29 14:04:03.885733 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.885716 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4"} err="failed to get container status \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": rpc error: code = NotFound desc = could not find container \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": container with ID starting with c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4 not found: ID does not exist" Jul 29 14:04:03.885778 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.885735 2549 scope.go:117] "RemoveContainer" containerID="5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" Jul 29 14:04:03.885948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.885928 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5"} err="failed to get container status \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": rpc error: code = NotFound desc = could not find container \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": container with ID starting with 5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5 not found: ID does not exist" Jul 29 14:04:03.885991 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.885949 2549 scope.go:117] "RemoveContainer" containerID="f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" Jul 29 14:04:03.886169 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.886150 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721"} err="failed to get container status \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": rpc error: code = NotFound desc = could not find container \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": container with ID starting with f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721 not found: ID does not exist" Jul 29 14:04:03.886238 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.886170 2549 scope.go:117] "RemoveContainer" containerID="528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" Jul 29 14:04:03.886479 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.886457 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007"} err="failed to get container status \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": rpc error: code = NotFound desc = could not find container \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": container with ID starting with 528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007 not found: ID does not exist" Jul 29 14:04:03.886532 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.886479 2549 scope.go:117] "RemoveContainer" containerID="6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d" Jul 29 14:04:03.886699 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.886680 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d"} err="failed to get container status \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": rpc error: code = NotFound desc = could not find container \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": container with ID starting with 6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d not found: ID does not exist" Jul 29 14:04:03.886743 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.886700 2549 scope.go:117] "RemoveContainer" containerID="7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" Jul 29 14:04:03.886940 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.886921 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b"} err="failed to get container status \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": rpc error: code = NotFound desc = could not find container \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": container with ID starting with 7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b not found: ID does not exist" Jul 29 14:04:03.886998 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.886940 2549 scope.go:117] "RemoveContainer" containerID="4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" Jul 29 14:04:03.887144 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.887117 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5"} err="failed to get container status \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": rpc error: code = NotFound desc = could not find container \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": container with ID starting with 4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5 not found: ID does not exist" Jul 29 14:04:03.887144 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.887135 2549 scope.go:117] "RemoveContainer" containerID="c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" Jul 29 14:04:03.887377 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.887354 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4"} err="failed to get container status \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": rpc error: code = NotFound desc = could not find container \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": container with ID starting with c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4 not found: ID does not exist" Jul 29 14:04:03.887450 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.887377 2549 scope.go:117] "RemoveContainer" containerID="5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" Jul 29 14:04:03.887593 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.887569 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5"} err="failed to get container status \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": rpc error: code = NotFound desc = could not find container \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": container with ID starting with 5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5 not found: ID does not exist" Jul 29 14:04:03.887636 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.887592 2549 scope.go:117] "RemoveContainer" containerID="f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" Jul 29 14:04:03.887815 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.887797 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721"} err="failed to get container status \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": rpc error: code = NotFound desc = could not find container \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": container with ID starting with f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721 not found: ID does not exist" Jul 29 14:04:03.887871 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.887815 2549 scope.go:117] "RemoveContainer" containerID="528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" Jul 29 14:04:03.888013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.887993 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007"} err="failed to get container status \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": rpc error: code = NotFound desc = could not find container \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": container with ID starting with 528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007 not found: ID does not exist" Jul 29 14:04:03.888081 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.888015 2549 scope.go:117] "RemoveContainer" containerID="6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d" Jul 29 14:04:03.888245 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.888226 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d"} err="failed to get container status \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": rpc error: code = NotFound desc = could not find container \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": container with ID starting with 6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d not found: ID does not exist" Jul 29 14:04:03.888303 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.888245 2549 scope.go:117] "RemoveContainer" containerID="7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" Jul 29 14:04:03.888515 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.888488 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b"} err="failed to get container status \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": rpc error: code = NotFound desc = could not find container \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": container with ID starting with 7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b not found: ID does not exist" Jul 29 14:04:03.888515 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.888507 2549 scope.go:117] "RemoveContainer" containerID="4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" Jul 29 14:04:03.888746 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.888729 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5"} err="failed to get container status \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": rpc error: code = NotFound desc = could not find container \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": container with ID starting with 4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5 not found: ID does not exist" Jul 29 14:04:03.888813 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.888748 2549 scope.go:117] "RemoveContainer" containerID="c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" Jul 29 14:04:03.888974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.888954 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4"} err="failed to get container status \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": rpc error: code = NotFound desc = could not find container \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": container with ID starting with c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4 not found: ID does not exist" Jul 29 14:04:03.889042 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.888975 2549 scope.go:117] "RemoveContainer" containerID="5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" Jul 29 14:04:03.889207 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.889190 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5"} err="failed to get container status \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": rpc error: code = NotFound desc = could not find container \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": container with ID starting with 5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5 not found: ID does not exist" Jul 29 14:04:03.889251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.889207 2549 scope.go:117] "RemoveContainer" containerID="f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" Jul 29 14:04:03.889431 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.889414 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721"} err="failed to get container status \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": rpc error: code = NotFound desc = could not find container \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": container with ID starting with f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721 not found: ID does not exist" Jul 29 14:04:03.889501 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.889432 2549 scope.go:117] "RemoveContainer" containerID="528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" Jul 29 14:04:03.889661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.889644 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007"} err="failed to get container status \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": rpc error: code = NotFound desc = could not find container \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": container with ID starting with 528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007 not found: ID does not exist" Jul 29 14:04:03.889718 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.889662 2549 scope.go:117] "RemoveContainer" containerID="6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d" Jul 29 14:04:03.889866 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.889849 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d"} err="failed to get container status \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": rpc error: code = NotFound desc = could not find container \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": container with ID starting with 6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d not found: ID does not exist" Jul 29 14:04:03.889929 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.889867 2549 scope.go:117] "RemoveContainer" containerID="7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" Jul 29 14:04:03.890059 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890042 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b"} err="failed to get container status \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": rpc error: code = NotFound desc = could not find container \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": container with ID starting with 7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b not found: ID does not exist" Jul 29 14:04:03.890119 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890061 2549 scope.go:117] "RemoveContainer" containerID="4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" Jul 29 14:04:03.890273 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890258 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5"} err="failed to get container status \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": rpc error: code = NotFound desc = could not find container \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": container with ID starting with 4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5 not found: ID does not exist" Jul 29 14:04:03.890366 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890274 2549 scope.go:117] "RemoveContainer" containerID="c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" Jul 29 14:04:03.890510 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890494 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4"} err="failed to get container status \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": rpc error: code = NotFound desc = could not find container \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": container with ID starting with c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4 not found: ID does not exist" Jul 29 14:04:03.890554 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890511 2549 scope.go:117] "RemoveContainer" containerID="5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" Jul 29 14:04:03.890705 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890687 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5"} err="failed to get container status \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": rpc error: code = NotFound desc = could not find container \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": container with ID starting with 5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5 not found: ID does not exist" Jul 29 14:04:03.890767 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890707 2549 scope.go:117] "RemoveContainer" containerID="f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" Jul 29 14:04:03.890909 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890894 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721"} err="failed to get container status \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": rpc error: code = NotFound desc = could not find container \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": container with ID starting with f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721 not found: ID does not exist" Jul 29 14:04:03.890909 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.890908 2549 scope.go:117] "RemoveContainer" containerID="528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" Jul 29 14:04:03.891094 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.891077 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007"} err="failed to get container status \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": rpc error: code = NotFound desc = could not find container \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": container with ID starting with 528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007 not found: ID does not exist" Jul 29 14:04:03.891159 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.891095 2549 scope.go:117] "RemoveContainer" containerID="6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d" Jul 29 14:04:03.891296 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.891279 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d"} err="failed to get container status \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": rpc error: code = NotFound desc = could not find container \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": container with ID starting with 6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d not found: ID does not exist" Jul 29 14:04:03.891432 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.891297 2549 scope.go:117] "RemoveContainer" containerID="7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b" Jul 29 14:04:03.891669 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.891653 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b"} err="failed to get container status \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": rpc error: code = NotFound desc = could not find container \"7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b\": container with ID starting with 7b3f2387ece7523d6ad9ac734a50f4338e32c4c01a401d7933fb410aa8f29f0b not found: ID does not exist" Jul 29 14:04:03.891731 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.891669 2549 scope.go:117] "RemoveContainer" containerID="4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5" Jul 29 14:04:03.891883 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.891865 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5"} err="failed to get container status \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": rpc error: code = NotFound desc = could not find container \"4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5\": container with ID starting with 4e20bf3e2097732bf8685f579a9554d21bc6403ee602ba81109b0c1c4023cfd5 not found: ID does not exist" Jul 29 14:04:03.891927 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.891886 2549 scope.go:117] "RemoveContainer" containerID="c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4" Jul 29 14:04:03.892065 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.892045 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4"} err="failed to get container status \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": rpc error: code = NotFound desc = could not find container \"c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4\": container with ID starting with c386adb9fa8bbbceafa1285eb3cd62965cf3be977eb887e4d7f8f8d2da2be1b4 not found: ID does not exist" Jul 29 14:04:03.892125 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.892066 2549 scope.go:117] "RemoveContainer" containerID="5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5" Jul 29 14:04:03.892278 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.892263 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5"} err="failed to get container status \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": rpc error: code = NotFound desc = could not find container \"5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5\": container with ID starting with 5328cfa7d465c9ff7a1669c7499270be1bfb2076b8eb1c4e3d3c6c338a04a4f5 not found: ID does not exist" Jul 29 14:04:03.892342 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.892278 2549 scope.go:117] "RemoveContainer" containerID="f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721" Jul 29 14:04:03.892530 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.892511 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721"} err="failed to get container status \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": rpc error: code = NotFound desc = could not find container \"f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721\": container with ID starting with f89ba546886cb550a84840fed3065ce8691be5e5d27e706d3772c3e61e1d1721 not found: ID does not exist" Jul 29 14:04:03.892571 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.892531 2549 scope.go:117] "RemoveContainer" containerID="528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007" Jul 29 14:04:03.892769 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.892750 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007"} err="failed to get container status \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": rpc error: code = NotFound desc = could not find container \"528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007\": container with ID starting with 528df38a88f2c05fb56d4191dd1d3e863e11c50daaa92b9786a9a4d905fce007 not found: ID does not exist" Jul 29 14:04:03.892769 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.892768 2549 scope.go:117] "RemoveContainer" containerID="6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d" Jul 29 14:04:03.892978 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.892963 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d"} err="failed to get container status \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": rpc error: code = NotFound desc = could not find container \"6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d\": container with ID starting with 6ade30ec47b58868e9a6350509fb6d3f4a6a8cb7ec87d0d0e2e853a148a8fa4d not found: ID does not exist" Jul 29 14:04:03.920685 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.920643 2549 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-config-out\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.920856 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.920776 2549 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-metrics-client-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.920930 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.920902 2549 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-grpc-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.920930 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.920920 2549 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-tls-assets\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921042 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.920947 2549 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921042 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.920961 2549 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921042 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.920974 2549 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-db\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921042 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.920984 2549 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921042 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.920993 2549 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921042 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921002 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqgl7\" (UniqueName: \"kubernetes.io/projected/a7605399-8eb1-4a7e-a070-c8bb681c4445-kube-api-access-fqgl7\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921042 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921038 2549 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921057 2549 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-kube-rbac-proxy\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921073 2549 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-config\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921089 2549 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-metrics-client-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921104 2549 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-web-config\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921118 2549 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921133 2549 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7605399-8eb1-4a7e-a070-c8bb681c4445-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921147 2549 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:03.921302 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:03.921161 2549 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7605399-8eb1-4a7e-a070-c8bb681c4445-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:04:04.150836 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.150797 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 14:04:04.154589 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.154560 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 14:04:04.180743 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.180709 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 14:04:04.181030 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181016 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy-web" Jul 29 14:04:04.181095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181032 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy-web" Jul 29 14:04:04.181095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181042 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy" Jul 29 14:04:04.181095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181048 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy" Jul 29 14:04:04.181095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181058 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="thanos-sidecar" Jul 29 14:04:04.181095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181066 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="thanos-sidecar" Jul 29 14:04:04.181095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181072 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy-thanos" Jul 29 14:04:04.181095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181078 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy-thanos" Jul 29 14:04:04.181095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181086 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="init-config-reloader" Jul 29 14:04:04.181095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181091 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="init-config-reloader" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181099 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="prometheus" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181105 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="prometheus" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181117 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="config-reloader" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181122 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="config-reloader" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181168 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="config-reloader" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181176 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy-web" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181183 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy-thanos" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181189 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="thanos-sidecar" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181194 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="kube-rbac-proxy" Jul 29 14:04:04.181370 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.181200 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" containerName="prometheus" Jul 29 14:04:04.185141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.185118 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.187761 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.187736 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 29 14:04:04.187761 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.187746 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 29 14:04:04.187974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.187779 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 29 14:04:04.187974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.187913 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 29 14:04:04.188089 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.187999 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 29 14:04:04.188089 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.188022 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 29 14:04:04.188217 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.188170 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 29 14:04:04.188271 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.188259 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 29 14:04:04.188758 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.188731 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 29 14:04:04.188758 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.188747 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r6b8z\"" Jul 29 14:04:04.188758 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.188754 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 29 14:04:04.188965 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.188802 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aspjhen8bhjqh\"" Jul 29 14:04:04.188965 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.188755 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 29 14:04:04.189354 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.189339 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 29 14:04:04.190888 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.190864 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 29 14:04:04.193687 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.193667 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 29 14:04:04.200356 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.200326 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 14:04:04.323672 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323631 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.323672 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323692 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.323903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323721 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.323903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323750 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.323903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323766 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxggr\" (UniqueName: \"kubernetes.io/projected/67cfc846-94af-45ca-92ac-0879dd97b970-kube-api-access-xxggr\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.323903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323814 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67cfc846-94af-45ca-92ac-0879dd97b970-config-out\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.323903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323843 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.323903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323862 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-config\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.323903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323877 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67cfc846-94af-45ca-92ac-0879dd97b970-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.323903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323906 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324143 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.323986 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-web-config\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324143 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.324022 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324143 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.324053 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324143 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.324072 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324143 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.324089 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324143 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.324128 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324326 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.324147 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324326 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.324169 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324326 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.324188 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.324326 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.324204 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.424724 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.424627 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.424724 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.424670 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.424724 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.424692 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.424874 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.424916 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.424946 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.424971 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425000 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425053 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425075 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425118 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425141 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxggr\" (UniqueName: \"kubernetes.io/projected/67cfc846-94af-45ca-92ac-0879dd97b970-kube-api-access-xxggr\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425178 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67cfc846-94af-45ca-92ac-0879dd97b970-config-out\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425208 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425224 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425235 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-config\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425261 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67cfc846-94af-45ca-92ac-0879dd97b970-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425329 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425379 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-web-config\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425405 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425452 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425621 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425684 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425792 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.425974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.425872 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.426403 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.426233 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.428254 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.427958 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.428254 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.428083 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.428254 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.428102 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.428526 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.428334 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.428872 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.428844 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.429024 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.429003 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.429141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.429111 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.429549 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.429524 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.429641 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.429594 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-config\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.430466 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.430444 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67cfc846-94af-45ca-92ac-0879dd97b970-web-config\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.430682 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.430664 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67cfc846-94af-45ca-92ac-0879dd97b970-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.430868 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.430852 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67cfc846-94af-45ca-92ac-0879dd97b970-config-out\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.431599 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.431581 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/67cfc846-94af-45ca-92ac-0879dd97b970-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.434110 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.434089 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxggr\" (UniqueName: \"kubernetes.io/projected/67cfc846-94af-45ca-92ac-0879dd97b970-kube-api-access-xxggr\") pod \"prometheus-k8s-0\" (UID: \"67cfc846-94af-45ca-92ac-0879dd97b970\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.496605 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.496474 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:04:04.643126 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.643100 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 14:04:04.645271 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:04:04.645240 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cfc846_94af_45ca_92ac_0879dd97b970.slice/crio-b27df938486e08bacd957c6fd23cb3a10e94b5c219ebbeb7ba510b72ac8c7826 WatchSource:0}: Error finding container b27df938486e08bacd957c6fd23cb3a10e94b5c219ebbeb7ba510b72ac8c7826: Status 404 returned error can't find the container with id b27df938486e08bacd957c6fd23cb3a10e94b5c219ebbeb7ba510b72ac8c7826 Jul 29 14:04:04.833530 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.833490 2549 generic.go:356] "Generic (PLEG): container finished" podID="67cfc846-94af-45ca-92ac-0879dd97b970" containerID="079f1291fb59e6c613aaae27c0a11261b8fecd3b0a0d5754670d80a463d1fbed" exitCode=0 Jul 29 14:04:04.833736 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.833576 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67cfc846-94af-45ca-92ac-0879dd97b970","Type":"ContainerDied","Data":"079f1291fb59e6c613aaae27c0a11261b8fecd3b0a0d5754670d80a463d1fbed"} Jul 29 14:04:04.833736 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.833664 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67cfc846-94af-45ca-92ac-0879dd97b970","Type":"ContainerStarted","Data":"b27df938486e08bacd957c6fd23cb3a10e94b5c219ebbeb7ba510b72ac8c7826"} Jul 29 14:04:04.871803 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:04.871775 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7605399-8eb1-4a7e-a070-c8bb681c4445" path="/var/lib/kubelet/pods/a7605399-8eb1-4a7e-a070-c8bb681c4445/volumes" Jul 29 14:04:05.842232 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:05.842150 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67cfc846-94af-45ca-92ac-0879dd97b970","Type":"ContainerStarted","Data":"d3a27983cc3cc1e6cea5fea9a25803d8bac549e2e5cb500c215dd20bf3b5c6a2"} Jul 29 14:04:05.842635 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:05.842237 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67cfc846-94af-45ca-92ac-0879dd97b970","Type":"ContainerStarted","Data":"d3066a9025561e28eb749ca6dc098bfd2e3ef7c7e10499685d78f889f8f6a813"} Jul 29 14:04:05.842635 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:05.842276 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67cfc846-94af-45ca-92ac-0879dd97b970","Type":"ContainerStarted","Data":"7c0da2552a7db1c681088c37ec22c9e3b7d76a3841b37c988f283d3b22f2c88f"} Jul 29 14:04:05.842635 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:05.842339 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67cfc846-94af-45ca-92ac-0879dd97b970","Type":"ContainerStarted","Data":"cb7918a33581e7cb48db8a2d3b2f12784cfe31877601cbafe4d8a8864c332cb1"} Jul 29 14:04:05.842635 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:05.842381 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67cfc846-94af-45ca-92ac-0879dd97b970","Type":"ContainerStarted","Data":"73d98386f2a1da1356161123b5542c37db245e950bbec0537b26c8b1add16101"} Jul 29 14:04:05.842635 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:05.842415 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"67cfc846-94af-45ca-92ac-0879dd97b970","Type":"ContainerStarted","Data":"c68a9b90ebb8c045e01278a62ef38ecc8356ecab7810bae1c31419817a6b7124"} Jul 29 14:04:05.869811 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:05.869738 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.849824382 podStartE2EDuration="1.869713271s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.888889ms" isStatefulPod=false podCreationTimestamp="2026-07-29 14:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:04:05.868385139 +0000 UTC m=+171.520978973" watchObservedRunningTime="2026-07-29 14:04:05.869713271 +0000 UTC m=+171.522307107" Jul 29 14:04:09.497047 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:04:09.496996 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:05:04.497172 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:05:04.497124 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:05:04.515372 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:05:04.515336 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:05:05.098545 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:05:05.098513 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 14:06:14.736497 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:14.736461 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:06:14.748926 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:14.748894 2549 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 14:06:14.751639 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:14.751616 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:06:21.596593 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.596555 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m"] Jul 29 14:06:21.599050 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.599030 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.601804 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.601771 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 14:06:21.602009 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.601964 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 14:06:21.602208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.602194 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 14:06:21.602255 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.602224 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2g49j\"" Jul 29 14:06:21.602507 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.602490 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 14:06:21.620644 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.620609 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m"] Jul 29 14:06:21.712885 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.712844 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fc5bb27-b105-441d-b3eb-33a1eea756e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.713061 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.712902 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc5bb27-b105-441d-b3eb-33a1eea756e8-tmp\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.713061 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.712964 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5fc5bb27-b105-441d-b3eb-33a1eea756e8-manifests\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.713061 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.713047 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5fc5bb27-b105-441d-b3eb-33a1eea756e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.713159 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.713067 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcgw\" (UniqueName: \"kubernetes.io/projected/5fc5bb27-b105-441d-b3eb-33a1eea756e8-kube-api-access-5xcgw\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.813744 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.813699 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc5bb27-b105-441d-b3eb-33a1eea756e8-tmp\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.813952 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.813755 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5fc5bb27-b105-441d-b3eb-33a1eea756e8-manifests\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.813952 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.813813 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5fc5bb27-b105-441d-b3eb-33a1eea756e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.813952 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.813839 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcgw\" (UniqueName: \"kubernetes.io/projected/5fc5bb27-b105-441d-b3eb-33a1eea756e8-kube-api-access-5xcgw\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.813952 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.813883 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fc5bb27-b105-441d-b3eb-33a1eea756e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.814300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.814274 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5fc5bb27-b105-441d-b3eb-33a1eea756e8-manifests\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.816214 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.816189 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc5bb27-b105-441d-b3eb-33a1eea756e8-tmp\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.816464 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.816445 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fc5bb27-b105-441d-b3eb-33a1eea756e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.816581 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.816559 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5fc5bb27-b105-441d-b3eb-33a1eea756e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.825915 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.825879 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcgw\" (UniqueName: \"kubernetes.io/projected/5fc5bb27-b105-441d-b3eb-33a1eea756e8-kube-api-access-5xcgw\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m\" (UID: \"5fc5bb27-b105-441d-b3eb-33a1eea756e8\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:21.910001 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:21.909872 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:22.052344 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:22.052263 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m"] Jul 29 14:06:22.058467 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:06:22.058427 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc5bb27_b105_441d_b3eb_33a1eea756e8.slice/crio-193b383e2d1fa87210aa91d3f49d575ac355aa922f7c19d627e48f47840d1f8c WatchSource:0}: Error finding container 193b383e2d1fa87210aa91d3f49d575ac355aa922f7c19d627e48f47840d1f8c: Status 404 returned error can't find the container with id 193b383e2d1fa87210aa91d3f49d575ac355aa922f7c19d627e48f47840d1f8c Jul 29 14:06:22.060198 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:22.060182 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:06:22.392133 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:22.392097 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" event={"ID":"5fc5bb27-b105-441d-b3eb-33a1eea756e8","Type":"ContainerStarted","Data":"193b383e2d1fa87210aa91d3f49d575ac355aa922f7c19d627e48f47840d1f8c"} Jul 29 14:06:25.408827 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:25.408781 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" event={"ID":"5fc5bb27-b105-441d-b3eb-33a1eea756e8","Type":"ContainerStarted","Data":"28fdd7eed813e6cfe96d79bc56c9e103479c55bf3dbb4a583dfe0880126112b9"} Jul 29 14:06:27.419901 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:27.419863 2549 generic.go:356] "Generic (PLEG): container finished" podID="5fc5bb27-b105-441d-b3eb-33a1eea756e8" containerID="28fdd7eed813e6cfe96d79bc56c9e103479c55bf3dbb4a583dfe0880126112b9" exitCode=0 Jul 29 14:06:27.420357 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:27.419917 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" event={"ID":"5fc5bb27-b105-441d-b3eb-33a1eea756e8","Type":"ContainerDied","Data":"28fdd7eed813e6cfe96d79bc56c9e103479c55bf3dbb4a583dfe0880126112b9"} Jul 29 14:06:28.426070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:28.425970 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" event={"ID":"5fc5bb27-b105-441d-b3eb-33a1eea756e8","Type":"ContainerStarted","Data":"2a9f6371bae29f5da48f841eadc9ecdb8d2c9558104cf10da54f823c3ae34d7e"} Jul 29 14:06:28.426513 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:28.426354 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:28.447099 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:28.447034 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" podStartSLOduration=2.42488501 podStartE2EDuration="7.447011761s" totalImagesPullingTime="3.845612575s" totalInitContainerRuntime="1.176514176s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:06:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:06:28.445837743 +0000 UTC m=+314.098431599" watchObservedRunningTime="2026-07-29 14:06:28.447011761 +0000 UTC m=+314.099605598" Jul 29 14:06:39.433962 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:39.433929 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m" Jul 29 14:06:51.732514 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.732477 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf"] Jul 29 14:06:51.738720 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.738695 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.743285 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.743257 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:06:51.743854 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.743831 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 14:06:51.744036 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.744017 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 29 14:06:51.744525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.744503 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kpql8\"" Jul 29 14:06:51.744779 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.744540 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 29 14:06:51.744882 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.744683 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 29 14:06:51.745084 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.745058 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf"] Jul 29 14:06:51.890975 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.890934 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5fbl\" (UniqueName: \"kubernetes.io/projected/a0866617-d190-4c22-85eb-cda2a4b0ec0e-kube-api-access-h5fbl\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.891195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.891028 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0866617-d190-4c22-85eb-cda2a4b0ec0e-manager-config\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.891195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.891055 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0866617-d190-4c22-85eb-cda2a4b0ec0e-metrics-cert\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.891195 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.891074 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0866617-d190-4c22-85eb-cda2a4b0ec0e-cert\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.992050 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.991947 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0866617-d190-4c22-85eb-cda2a4b0ec0e-manager-config\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.992050 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.991984 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0866617-d190-4c22-85eb-cda2a4b0ec0e-metrics-cert\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.992050 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.992002 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0866617-d190-4c22-85eb-cda2a4b0ec0e-cert\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.992283 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.992068 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5fbl\" (UniqueName: \"kubernetes.io/projected/a0866617-d190-4c22-85eb-cda2a4b0ec0e-kube-api-access-h5fbl\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.992705 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.992678 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0866617-d190-4c22-85eb-cda2a4b0ec0e-manager-config\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.994749 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.994725 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0866617-d190-4c22-85eb-cda2a4b0ec0e-metrics-cert\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:51.994897 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:51.994877 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0866617-d190-4c22-85eb-cda2a4b0ec0e-cert\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:52.003766 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.003735 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5fbl\" (UniqueName: \"kubernetes.io/projected/a0866617-d190-4c22-85eb-cda2a4b0ec0e-kube-api-access-h5fbl\") pod \"lws-controller-manager-f9f5fc96c-f7qsf\" (UID: \"a0866617-d190-4c22-85eb-cda2a4b0ec0e\") " pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:52.050630 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.050589 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:52.196489 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.196435 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf"] Jul 29 14:06:52.199144 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:06:52.199106 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0866617_d190_4c22_85eb_cda2a4b0ec0e.slice/crio-84b129aee5ee76d1cccf6dc5abd6bc1255fad7852e3d5cd518e6e4830681bb94 WatchSource:0}: Error finding container 84b129aee5ee76d1cccf6dc5abd6bc1255fad7852e3d5cd518e6e4830681bb94: Status 404 returned error can't find the container with id 84b129aee5ee76d1cccf6dc5abd6bc1255fad7852e3d5cd518e6e4830681bb94 Jul 29 14:06:52.241306 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.241251 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp"] Jul 29 14:06:52.244752 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.244677 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.247801 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.247765 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 14:06:52.247801 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.247787 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 14:06:52.247801 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.247801 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 14:06:52.248070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.247765 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 14:06:52.248123 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.248110 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 14:06:52.248419 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.248406 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 14:06:52.251475 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.251455 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5hpqr\"" Jul 29 14:06:52.266464 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.262874 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp"] Jul 29 14:06:52.396279 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.396237 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce73abf6-8088-4d56-9338-57d2095a1ad6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.396525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.396297 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.396525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.396352 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gtgq\" (UniqueName: \"kubernetes.io/projected/ce73abf6-8088-4d56-9338-57d2095a1ad6-kube-api-access-6gtgq\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.396525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.396374 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.396525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.396394 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.396525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.396418 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce73abf6-8088-4d56-9338-57d2095a1ad6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.396525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.396444 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.497187 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.497081 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.497187 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.497131 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce73abf6-8088-4d56-9338-57d2095a1ad6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.497187 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.497160 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.497501 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.497210 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce73abf6-8088-4d56-9338-57d2095a1ad6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.497501 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.497251 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.497501 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.497285 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gtgq\" (UniqueName: \"kubernetes.io/projected/ce73abf6-8088-4d56-9338-57d2095a1ad6-kube-api-access-6gtgq\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.497501 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.497349 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.498124 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.498092 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.499880 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.499844 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.500408 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.500384 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce73abf6-8088-4d56-9338-57d2095a1ad6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.500503 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.500386 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce73abf6-8088-4d56-9338-57d2095a1ad6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.500503 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.500416 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.512039 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.512010 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce73abf6-8088-4d56-9338-57d2095a1ad6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.512176 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.512010 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gtgq\" (UniqueName: \"kubernetes.io/projected/ce73abf6-8088-4d56-9338-57d2095a1ad6-kube-api-access-6gtgq\") pod \"istiod-openshift-gateway-5677ccb49b-79nlp\" (UID: \"ce73abf6-8088-4d56-9338-57d2095a1ad6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.535901 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.535861 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" event={"ID":"a0866617-d190-4c22-85eb-cda2a4b0ec0e","Type":"ContainerStarted","Data":"84b129aee5ee76d1cccf6dc5abd6bc1255fad7852e3d5cd518e6e4830681bb94"} Jul 29 14:06:52.558299 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.558238 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:52.707165 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:52.707126 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp"] Jul 29 14:06:52.710400 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:06:52.710364 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce73abf6_8088_4d56_9338_57d2095a1ad6.slice/crio-c33edc3253f4e2df9b6965e39cd0813a7079be6418416d64c360466408235a53 WatchSource:0}: Error finding container c33edc3253f4e2df9b6965e39cd0813a7079be6418416d64c360466408235a53: Status 404 returned error can't find the container with id c33edc3253f4e2df9b6965e39cd0813a7079be6418416d64c360466408235a53 Jul 29 14:06:53.549260 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:53.549093 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" event={"ID":"ce73abf6-8088-4d56-9338-57d2095a1ad6","Type":"ContainerStarted","Data":"c33edc3253f4e2df9b6965e39cd0813a7079be6418416d64c360466408235a53"} Jul 29 14:06:56.147356 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:56.147303 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 29 14:06:56.147799 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:56.147406 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 29 14:06:56.567838 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:56.567800 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" event={"ID":"a0866617-d190-4c22-85eb-cda2a4b0ec0e","Type":"ContainerStarted","Data":"6ae772de70c480b6cb92d72693aaceb3fc848a5ad3a083554b88829c052975c1"} Jul 29 14:06:56.568093 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:56.567899 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:06:56.569670 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:56.569635 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" event={"ID":"ce73abf6-8088-4d56-9338-57d2095a1ad6","Type":"ContainerStarted","Data":"e5b55149a38139ea240bb032a61d2da808f10c88cb56149fbb73533ae8bd9269"} Jul 29 14:06:56.569843 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:56.569685 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:06:56.584218 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:56.584163 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" podStartSLOduration=1.639877422 podStartE2EDuration="5.584143883s" totalImagesPullingTime="3.944266461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:06:56.583282077 +0000 UTC m=+342.235875913" watchObservedRunningTime="2026-07-29 14:06:56.584143883 +0000 UTC m=+342.236737717" Jul 29 14:06:56.603607 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:56.603533 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" podStartSLOduration=1.169004169 podStartE2EDuration="4.60350658s" totalImagesPullingTime="3.434502411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:06:56.60277565 +0000 UTC m=+342.255369508" watchObservedRunningTime="2026-07-29 14:06:56.60350658 +0000 UTC m=+342.256100415" Jul 29 14:06:57.575750 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:06:57.575716 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-79nlp" Jul 29 14:07:07.576989 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:07:07.576903 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f9f5fc96c-f7qsf" Jul 29 14:08:46.452846 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.452760 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn"] Jul 29 14:08:46.455021 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.455002 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" Jul 29 14:08:46.457863 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.457835 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 14:08:46.458809 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.458789 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 14:08:46.458928 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.458871 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-skfhd\"" Jul 29 14:08:46.467163 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.467131 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn"] Jul 29 14:08:46.487663 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.487621 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xglj\" (UniqueName: \"kubernetes.io/projected/57e4ef31-edff-46ff-aebe-944de3c05fe9-kube-api-access-7xglj\") pod \"limitador-operator-controller-manager-859d7b78bd-qbhjn\" (UID: \"57e4ef31-edff-46ff-aebe-944de3c05fe9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" Jul 29 14:08:46.588584 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.588541 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xglj\" (UniqueName: \"kubernetes.io/projected/57e4ef31-edff-46ff-aebe-944de3c05fe9-kube-api-access-7xglj\") pod \"limitador-operator-controller-manager-859d7b78bd-qbhjn\" (UID: \"57e4ef31-edff-46ff-aebe-944de3c05fe9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" Jul 29 14:08:46.602757 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.602725 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xglj\" (UniqueName: \"kubernetes.io/projected/57e4ef31-edff-46ff-aebe-944de3c05fe9-kube-api-access-7xglj\") pod \"limitador-operator-controller-manager-859d7b78bd-qbhjn\" (UID: \"57e4ef31-edff-46ff-aebe-944de3c05fe9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" Jul 29 14:08:46.766653 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.766617 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" Jul 29 14:08:46.922334 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:46.922274 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn"] Jul 29 14:08:46.925844 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:08:46.925811 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e4ef31_edff_46ff_aebe_944de3c05fe9.slice/crio-21b39567b210162ab070a155d770007641d7784e019d3b4b65d950caf84ba1e6 WatchSource:0}: Error finding container 21b39567b210162ab070a155d770007641d7784e019d3b4b65d950caf84ba1e6: Status 404 returned error can't find the container with id 21b39567b210162ab070a155d770007641d7784e019d3b4b65d950caf84ba1e6 Jul 29 14:08:47.080874 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:47.080782 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" event={"ID":"57e4ef31-edff-46ff-aebe-944de3c05fe9","Type":"ContainerStarted","Data":"21b39567b210162ab070a155d770007641d7784e019d3b4b65d950caf84ba1e6"} Jul 29 14:08:50.096553 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:50.096512 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" event={"ID":"57e4ef31-edff-46ff-aebe-944de3c05fe9","Type":"ContainerStarted","Data":"532a7a82bb3ab7233deb4c8cef0fd2df050d623ecccc686fe03224a52a32fbe5"} Jul 29 14:08:50.097044 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:50.096651 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" Jul 29 14:08:50.116337 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:08:50.116246 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" podStartSLOduration=2.012981004 podStartE2EDuration="4.116226858s" totalImagesPullingTime="2.103245854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:08:50.115700594 +0000 UTC m=+455.768294472" watchObservedRunningTime="2026-07-29 14:08:50.116226858 +0000 UTC m=+455.768820693" Jul 29 14:09:01.103858 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:09:01.103825 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qbhjn" Jul 29 14:10:21.191412 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.191288 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-797f89dfc7-9n6sq"] Jul 29 14:10:21.193581 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.193561 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:21.196104 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.196080 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 14:10:21.196201 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.196172 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 14:10:21.196258 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.196230 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c8bls\"" Jul 29 14:10:21.206004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.205973 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-797f89dfc7-9n6sq"] Jul 29 14:10:21.262070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.262024 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3edc2dfd-4019-481f-9455-45686c8e78cf-webhook-cert\") pod \"maas-controller-797f89dfc7-9n6sq\" (UID: \"3edc2dfd-4019-481f-9455-45686c8e78cf\") " pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:21.262562 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.262540 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpg6\" (UniqueName: \"kubernetes.io/projected/3edc2dfd-4019-481f-9455-45686c8e78cf-kube-api-access-xjpg6\") pod \"maas-controller-797f89dfc7-9n6sq\" (UID: \"3edc2dfd-4019-481f-9455-45686c8e78cf\") " pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:21.363679 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.363636 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpg6\" (UniqueName: \"kubernetes.io/projected/3edc2dfd-4019-481f-9455-45686c8e78cf-kube-api-access-xjpg6\") pod \"maas-controller-797f89dfc7-9n6sq\" (UID: \"3edc2dfd-4019-481f-9455-45686c8e78cf\") " pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:21.363896 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.363780 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3edc2dfd-4019-481f-9455-45686c8e78cf-webhook-cert\") pod \"maas-controller-797f89dfc7-9n6sq\" (UID: \"3edc2dfd-4019-481f-9455-45686c8e78cf\") " pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:21.366906 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.366877 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3edc2dfd-4019-481f-9455-45686c8e78cf-webhook-cert\") pod \"maas-controller-797f89dfc7-9n6sq\" (UID: \"3edc2dfd-4019-481f-9455-45686c8e78cf\") " pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:21.367248 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.367216 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d87754bdc-s6xnv"] Jul 29 14:10:21.375158 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.375131 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:21.377074 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.377043 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpg6\" (UniqueName: \"kubernetes.io/projected/3edc2dfd-4019-481f-9455-45686c8e78cf-kube-api-access-xjpg6\") pod \"maas-controller-797f89dfc7-9n6sq\" (UID: \"3edc2dfd-4019-481f-9455-45686c8e78cf\") " pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:21.378996 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.378970 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d87754bdc-s6xnv"] Jul 29 14:10:21.464484 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.464374 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a9362c5-046d-4f83-941e-be629219253a-webhook-cert\") pod \"maas-controller-d87754bdc-s6xnv\" (UID: \"5a9362c5-046d-4f83-941e-be629219253a\") " pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:21.464484 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.464448 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xpl\" (UniqueName: \"kubernetes.io/projected/5a9362c5-046d-4f83-941e-be629219253a-kube-api-access-q7xpl\") pod \"maas-controller-d87754bdc-s6xnv\" (UID: \"5a9362c5-046d-4f83-941e-be629219253a\") " pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:21.505149 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.505117 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:21.565577 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.565542 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a9362c5-046d-4f83-941e-be629219253a-webhook-cert\") pod \"maas-controller-d87754bdc-s6xnv\" (UID: \"5a9362c5-046d-4f83-941e-be629219253a\") " pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:21.565788 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.565595 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xpl\" (UniqueName: \"kubernetes.io/projected/5a9362c5-046d-4f83-941e-be629219253a-kube-api-access-q7xpl\") pod \"maas-controller-d87754bdc-s6xnv\" (UID: \"5a9362c5-046d-4f83-941e-be629219253a\") " pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:21.568947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.568883 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a9362c5-046d-4f83-941e-be629219253a-webhook-cert\") pod \"maas-controller-d87754bdc-s6xnv\" (UID: \"5a9362c5-046d-4f83-941e-be629219253a\") " pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:21.575830 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.575797 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xpl\" (UniqueName: \"kubernetes.io/projected/5a9362c5-046d-4f83-941e-be629219253a-kube-api-access-q7xpl\") pod \"maas-controller-d87754bdc-s6xnv\" (UID: \"5a9362c5-046d-4f83-941e-be629219253a\") " pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:21.641907 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.641881 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-797f89dfc7-9n6sq"] Jul 29 14:10:21.645280 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:10:21.645241 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edc2dfd_4019_481f_9455_45686c8e78cf.slice/crio-a8f8d77ba29e8b00466e304aaafed63e1d3947e09c9529f9e78644f0eb505b3a WatchSource:0}: Error finding container a8f8d77ba29e8b00466e304aaafed63e1d3947e09c9529f9e78644f0eb505b3a: Status 404 returned error can't find the container with id a8f8d77ba29e8b00466e304aaafed63e1d3947e09c9529f9e78644f0eb505b3a Jul 29 14:10:21.694275 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.694229 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:21.836425 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:21.836393 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d87754bdc-s6xnv"] Jul 29 14:10:21.838484 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:10:21.838457 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9362c5_046d_4f83_941e_be629219253a.slice/crio-a96451f799142ee4424ae88249548893d85636624b3cdaa6aa707bd40e096c7d WatchSource:0}: Error finding container a96451f799142ee4424ae88249548893d85636624b3cdaa6aa707bd40e096c7d: Status 404 returned error can't find the container with id a96451f799142ee4424ae88249548893d85636624b3cdaa6aa707bd40e096c7d Jul 29 14:10:22.513774 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:22.513728 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-s6xnv" event={"ID":"5a9362c5-046d-4f83-941e-be629219253a","Type":"ContainerStarted","Data":"a96451f799142ee4424ae88249548893d85636624b3cdaa6aa707bd40e096c7d"} Jul 29 14:10:22.515460 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:22.515420 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" event={"ID":"3edc2dfd-4019-481f-9455-45686c8e78cf","Type":"ContainerStarted","Data":"a8f8d77ba29e8b00466e304aaafed63e1d3947e09c9529f9e78644f0eb505b3a"} Jul 29 14:10:24.526751 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:24.526715 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-s6xnv" event={"ID":"5a9362c5-046d-4f83-941e-be629219253a","Type":"ContainerStarted","Data":"fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca"} Jul 29 14:10:24.527342 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:24.526783 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:24.528336 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:24.528300 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" event={"ID":"3edc2dfd-4019-481f-9455-45686c8e78cf","Type":"ContainerStarted","Data":"3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538"} Jul 29 14:10:24.528405 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:24.528362 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:24.543926 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:24.543875 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d87754bdc-s6xnv" podStartSLOduration=1.2978746700000001 podStartE2EDuration="3.543856571s" totalImagesPullingTime="2.245981901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:10:24.543170714 +0000 UTC m=+550.195764548" watchObservedRunningTime="2026-07-29 14:10:24.543856571 +0000 UTC m=+550.196450404" Jul 29 14:10:24.562751 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:24.562672 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" podStartSLOduration=1.130058585 podStartE2EDuration="3.562642769s" totalImagesPullingTime="2.432584184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:10:24.560357469 +0000 UTC m=+550.212951295" watchObservedRunningTime="2026-07-29 14:10:24.562642769 +0000 UTC m=+550.215236603" Jul 29 14:10:35.538731 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:35.538691 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:10:35.539203 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:35.539075 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:35.598303 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:35.598264 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-797f89dfc7-9n6sq"] Jul 29 14:10:35.598555 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:35.598518 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" podUID="3edc2dfd-4019-481f-9455-45686c8e78cf" containerName="manager" containerID="cri-o://3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538" gracePeriod=30 Jul 29 14:10:35.853206 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:35.853178 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:36.008596 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.008550 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3edc2dfd-4019-481f-9455-45686c8e78cf-webhook-cert\") pod \"3edc2dfd-4019-481f-9455-45686c8e78cf\" (UID: \"3edc2dfd-4019-481f-9455-45686c8e78cf\") " Jul 29 14:10:36.008803 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.008719 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjpg6\" (UniqueName: \"kubernetes.io/projected/3edc2dfd-4019-481f-9455-45686c8e78cf-kube-api-access-xjpg6\") pod \"3edc2dfd-4019-481f-9455-45686c8e78cf\" (UID: \"3edc2dfd-4019-481f-9455-45686c8e78cf\") " Jul 29 14:10:36.011032 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.010992 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3edc2dfd-4019-481f-9455-45686c8e78cf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3edc2dfd-4019-481f-9455-45686c8e78cf" (UID: "3edc2dfd-4019-481f-9455-45686c8e78cf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:10:36.011167 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.011035 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edc2dfd-4019-481f-9455-45686c8e78cf-kube-api-access-xjpg6" (OuterVolumeSpecName: "kube-api-access-xjpg6") pod "3edc2dfd-4019-481f-9455-45686c8e78cf" (UID: "3edc2dfd-4019-481f-9455-45686c8e78cf"). InnerVolumeSpecName "kube-api-access-xjpg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:10:36.110573 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.110471 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjpg6\" (UniqueName: \"kubernetes.io/projected/3edc2dfd-4019-481f-9455-45686c8e78cf-kube-api-access-xjpg6\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:10:36.110573 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.110506 2549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3edc2dfd-4019-481f-9455-45686c8e78cf-webhook-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:10:36.586457 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.586421 2549 generic.go:356] "Generic (PLEG): container finished" podID="3edc2dfd-4019-481f-9455-45686c8e78cf" containerID="3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538" exitCode=0 Jul 29 14:10:36.586932 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.586470 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" Jul 29 14:10:36.586932 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.586479 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" event={"ID":"3edc2dfd-4019-481f-9455-45686c8e78cf","Type":"ContainerDied","Data":"3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538"} Jul 29 14:10:36.586932 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.586525 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-797f89dfc7-9n6sq" event={"ID":"3edc2dfd-4019-481f-9455-45686c8e78cf","Type":"ContainerDied","Data":"a8f8d77ba29e8b00466e304aaafed63e1d3947e09c9529f9e78644f0eb505b3a"} Jul 29 14:10:36.586932 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.586546 2549 scope.go:117] "RemoveContainer" containerID="3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538" Jul 29 14:10:36.595678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.595659 2549 scope.go:117] "RemoveContainer" containerID="3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538" Jul 29 14:10:36.595982 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:10:36.595961 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538\": container with ID starting with 3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538 not found: ID does not exist" containerID="3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538" Jul 29 14:10:36.596029 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.595993 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538"} err="failed to get container status \"3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538\": rpc error: code = NotFound desc = could not find container \"3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538\": container with ID starting with 3f40bd750dbd7f98d63a58a6a6da7f258ddd91b16cc41a21b6bc9b8b7fe1f538 not found: ID does not exist" Jul 29 14:10:36.609653 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.609583 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-797f89dfc7-9n6sq"] Jul 29 14:10:36.611318 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.611281 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-797f89dfc7-9n6sq"] Jul 29 14:10:36.867303 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:36.867211 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3edc2dfd-4019-481f-9455-45686c8e78cf" path="/var/lib/kubelet/pods/3edc2dfd-4019-481f-9455-45686c8e78cf/volumes" Jul 29 14:10:57.263714 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.263677 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-qswsb"] Jul 29 14:10:57.264210 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.264035 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3edc2dfd-4019-481f-9455-45686c8e78cf" containerName="manager" Jul 29 14:10:57.264210 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.264048 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3edc2dfd-4019-481f-9455-45686c8e78cf" containerName="manager" Jul 29 14:10:57.264210 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.264105 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3edc2dfd-4019-481f-9455-45686c8e78cf" containerName="manager" Jul 29 14:10:57.267363 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.267342 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:10:57.270130 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.270102 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kjsn4\"" Jul 29 14:10:57.270259 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.270141 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 29 14:10:57.279444 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.279414 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-qswsb"] Jul 29 14:10:57.307801 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.307754 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b59c1bd-570e-455f-89d7-d9e7a2856d63-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-qswsb\" (UID: \"0b59c1bd-570e-455f-89d7-d9e7a2856d63\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:10:57.308015 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.307828 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxhq\" (UniqueName: \"kubernetes.io/projected/0b59c1bd-570e-455f-89d7-d9e7a2856d63-kube-api-access-gcxhq\") pod \"payload-processing-5cc6cd444b-qswsb\" (UID: \"0b59c1bd-570e-455f-89d7-d9e7a2856d63\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:10:57.332749 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.332711 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk"] Jul 29 14:10:57.336141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.336120 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:10:57.349684 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.349652 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk"] Jul 29 14:10:57.409381 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.409338 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b59c1bd-570e-455f-89d7-d9e7a2856d63-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-qswsb\" (UID: \"0b59c1bd-570e-455f-89d7-d9e7a2856d63\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:10:57.409585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.409436 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76c7b8ab-30aa-4387-bc0a-3b267b46100a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-tmpvk\" (UID: \"76c7b8ab-30aa-4387-bc0a-3b267b46100a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:10:57.409585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.409467 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7b7h\" (UniqueName: \"kubernetes.io/projected/76c7b8ab-30aa-4387-bc0a-3b267b46100a-kube-api-access-k7b7h\") pod \"payload-pre-processing-687dbb6cf6-tmpvk\" (UID: \"76c7b8ab-30aa-4387-bc0a-3b267b46100a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:10:57.409585 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.409499 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxhq\" (UniqueName: \"kubernetes.io/projected/0b59c1bd-570e-455f-89d7-d9e7a2856d63-kube-api-access-gcxhq\") pod \"payload-processing-5cc6cd444b-qswsb\" (UID: \"0b59c1bd-570e-455f-89d7-d9e7a2856d63\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:10:57.410150 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.410125 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b59c1bd-570e-455f-89d7-d9e7a2856d63-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-qswsb\" (UID: \"0b59c1bd-570e-455f-89d7-d9e7a2856d63\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:10:57.420095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.420060 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxhq\" (UniqueName: \"kubernetes.io/projected/0b59c1bd-570e-455f-89d7-d9e7a2856d63-kube-api-access-gcxhq\") pod \"payload-processing-5cc6cd444b-qswsb\" (UID: \"0b59c1bd-570e-455f-89d7-d9e7a2856d63\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:10:57.510827 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.510780 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76c7b8ab-30aa-4387-bc0a-3b267b46100a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-tmpvk\" (UID: \"76c7b8ab-30aa-4387-bc0a-3b267b46100a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:10:57.510827 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.510822 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7b7h\" (UniqueName: \"kubernetes.io/projected/76c7b8ab-30aa-4387-bc0a-3b267b46100a-kube-api-access-k7b7h\") pod \"payload-pre-processing-687dbb6cf6-tmpvk\" (UID: \"76c7b8ab-30aa-4387-bc0a-3b267b46100a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:10:57.511551 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.511530 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76c7b8ab-30aa-4387-bc0a-3b267b46100a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-tmpvk\" (UID: \"76c7b8ab-30aa-4387-bc0a-3b267b46100a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:10:57.519949 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.519876 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7b7h\" (UniqueName: \"kubernetes.io/projected/76c7b8ab-30aa-4387-bc0a-3b267b46100a-kube-api-access-k7b7h\") pod \"payload-pre-processing-687dbb6cf6-tmpvk\" (UID: \"76c7b8ab-30aa-4387-bc0a-3b267b46100a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:10:57.578487 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.578453 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:10:57.647576 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.647543 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:10:57.739365 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.739301 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-qswsb"] Jul 29 14:10:57.743691 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:10:57.743661 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b59c1bd_570e_455f_89d7_d9e7a2856d63.slice/crio-7f01e68b17622d3fb55f1f98787d3806f8f6089160eaced56702703a815fa084 WatchSource:0}: Error finding container 7f01e68b17622d3fb55f1f98787d3806f8f6089160eaced56702703a815fa084: Status 404 returned error can't find the container with id 7f01e68b17622d3fb55f1f98787d3806f8f6089160eaced56702703a815fa084 Jul 29 14:10:57.811461 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:57.811349 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk"] Jul 29 14:10:57.814624 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:10:57.814595 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c7b8ab_30aa_4387_bc0a_3b267b46100a.slice/crio-3aa91941f9948d2c1018cfe84d0cb45e895ebafcc36e0e7347d6347adffe94a4 WatchSource:0}: Error finding container 3aa91941f9948d2c1018cfe84d0cb45e895ebafcc36e0e7347d6347adffe94a4: Status 404 returned error can't find the container with id 3aa91941f9948d2c1018cfe84d0cb45e895ebafcc36e0e7347d6347adffe94a4 Jul 29 14:10:58.698706 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:58.698602 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" event={"ID":"0b59c1bd-570e-455f-89d7-d9e7a2856d63","Type":"ContainerStarted","Data":"7f01e68b17622d3fb55f1f98787d3806f8f6089160eaced56702703a815fa084"} Jul 29 14:10:58.701769 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:10:58.701682 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" event={"ID":"76c7b8ab-30aa-4387-bc0a-3b267b46100a","Type":"ContainerStarted","Data":"3aa91941f9948d2c1018cfe84d0cb45e895ebafcc36e0e7347d6347adffe94a4"} Jul 29 14:11:00.713728 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:00.713685 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" event={"ID":"76c7b8ab-30aa-4387-bc0a-3b267b46100a","Type":"ContainerStarted","Data":"38b5517d706be28fc07cf610836d44b9363b7b8959fd16cf9f9fa1f3be749eee"} Jul 29 14:11:00.714210 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:00.713758 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:11:00.715231 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:00.715204 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" event={"ID":"0b59c1bd-570e-455f-89d7-d9e7a2856d63","Type":"ContainerStarted","Data":"09644d5d4746f6c437e31b359b6ea7e801afd5284134a44ce2168f762428d870"} Jul 29 14:11:00.715378 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:00.715267 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:11:00.730005 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:00.729947 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" podStartSLOduration=0.993759115 podStartE2EDuration="3.729925407s" totalImagesPullingTime="2.736166292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:11:00.729411521 +0000 UTC m=+586.382005358" watchObservedRunningTime="2026-07-29 14:11:00.729925407 +0000 UTC m=+586.382519242" Jul 29 14:11:00.746250 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:00.746189 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" podStartSLOduration=0.933025358 podStartE2EDuration="3.74616932s" totalImagesPullingTime="2.813143962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:11:00.744956715 +0000 UTC m=+586.397550564" watchObservedRunningTime="2026-07-29 14:11:00.74616932 +0000 UTC m=+586.398763153" Jul 29 14:11:11.722647 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:11.722611 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tmpvk" Jul 29 14:11:11.723046 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:11.722695 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-qswsb" Jul 29 14:11:14.774404 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:14.774373 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:11:14.777098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:14.777072 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:11:35.945162 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:35.945074 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5"] Jul 29 14:11:35.952393 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:35.952367 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:35.956106 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:35.956076 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 14:11:35.956271 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:35.956085 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 29 14:11:35.956271 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:35.956175 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 14:11:35.956271 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:35.956180 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nccq4\"" Jul 29 14:11:35.959284 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:35.959262 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5"] Jul 29 14:11:36.075257 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.075218 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.075502 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.075270 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.075502 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.075289 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.075502 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.075376 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.075502 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.075484 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.075696 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.075523 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh2k\" (UniqueName: \"kubernetes.io/projected/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-kube-api-access-phh2k\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.176830 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.176788 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.176830 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.176826 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.177099 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.176850 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.177099 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.176886 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.177099 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.176912 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh2k\" (UniqueName: \"kubernetes.io/projected/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-kube-api-access-phh2k\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.177099 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.177001 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.177344 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.177290 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.177344 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.177306 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.177434 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.177414 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.179174 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.179151 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.179609 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.179588 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.185959 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.185936 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh2k\" (UniqueName: \"kubernetes.io/projected/6ff07e17-7f6c-4ddf-b41d-7f736a663d71-kube-api-access-phh2k\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9gsn5\" (UID: \"6ff07e17-7f6c-4ddf-b41d-7f736a663d71\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.267229 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.267182 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:36.410992 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.410958 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5"] Jul 29 14:11:36.413791 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:11:36.413764 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff07e17_7f6c_4ddf_b41d_7f736a663d71.slice/crio-eb7c71b3a5799defe80f7e13919e539108c335766dde246de38be65b437faba9 WatchSource:0}: Error finding container eb7c71b3a5799defe80f7e13919e539108c335766dde246de38be65b437faba9: Status 404 returned error can't find the container with id eb7c71b3a5799defe80f7e13919e539108c335766dde246de38be65b437faba9 Jul 29 14:11:36.415133 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.415115 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:11:36.885141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:36.885096 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" event={"ID":"6ff07e17-7f6c-4ddf-b41d-7f736a663d71","Type":"ContainerStarted","Data":"eb7c71b3a5799defe80f7e13919e539108c335766dde246de38be65b437faba9"} Jul 29 14:11:37.891732 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:37.891631 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" event={"ID":"6ff07e17-7f6c-4ddf-b41d-7f736a663d71","Type":"ContainerStarted","Data":"72c3bc268e48c0b56ea73e561e431b1a541a95fa0686ad67b0d7d62cdedc2f6e"} Jul 29 14:11:37.910015 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:37.909959 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" podStartSLOduration=1.741598433 podStartE2EDuration="2.909938897s" totalImagesPullingTime="1.168340464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:11:37.908627381 +0000 UTC m=+623.561221216" watchObservedRunningTime="2026-07-29 14:11:37.909938897 +0000 UTC m=+623.562532728" Jul 29 14:11:46.267490 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:46.267451 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:46.267911 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:46.267542 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:46.280371 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:46.280332 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:46.948136 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:46.948106 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9gsn5" Jul 29 14:11:51.744284 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.744251 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4"] Jul 29 14:11:51.749111 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.749082 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.751668 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.751641 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 29 14:11:51.761968 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.761931 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4"] Jul 29 14:11:51.831588 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.831544 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c074b6d8-919f-41f6-b62b-ddc5fd80134b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.831780 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.831597 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.831780 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.831624 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.831780 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.831651 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2psbc\" (UniqueName: \"kubernetes.io/projected/c074b6d8-919f-41f6-b62b-ddc5fd80134b-kube-api-access-2psbc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.831780 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.831671 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.831780 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.831752 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.933097 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.933045 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.933097 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.933094 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.933432 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.933131 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2psbc\" (UniqueName: \"kubernetes.io/projected/c074b6d8-919f-41f6-b62b-ddc5fd80134b-kube-api-access-2psbc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.933432 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.933159 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.933432 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.933226 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.933586 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.933434 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c074b6d8-919f-41f6-b62b-ddc5fd80134b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.933648 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.933602 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.933648 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.933617 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.933754 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.933731 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.935671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.935642 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c074b6d8-919f-41f6-b62b-ddc5fd80134b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.935837 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.935817 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c074b6d8-919f-41f6-b62b-ddc5fd80134b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:51.941410 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:51.941383 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2psbc\" (UniqueName: \"kubernetes.io/projected/c074b6d8-919f-41f6-b62b-ddc5fd80134b-kube-api-access-2psbc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4\" (UID: \"c074b6d8-919f-41f6-b62b-ddc5fd80134b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:52.061635 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:52.061583 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:11:52.207052 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:52.207004 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4"] Jul 29 14:11:52.209178 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:11:52.209148 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc074b6d8_919f_41f6_b62b_ddc5fd80134b.slice/crio-b5228488729828cd5618f65ecfb2361fea4de453779013bff9ea2ee15eb0ebf2 WatchSource:0}: Error finding container b5228488729828cd5618f65ecfb2361fea4de453779013bff9ea2ee15eb0ebf2: Status 404 returned error can't find the container with id b5228488729828cd5618f65ecfb2361fea4de453779013bff9ea2ee15eb0ebf2 Jul 29 14:11:52.966198 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:52.966101 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" event={"ID":"c074b6d8-919f-41f6-b62b-ddc5fd80134b","Type":"ContainerStarted","Data":"e1e4d778d7605431d57e33ffd498da245ca28f47af1d4a71dbba1ee17b266b15"} Jul 29 14:11:52.966198 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:52.966160 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" event={"ID":"c074b6d8-919f-41f6-b62b-ddc5fd80134b","Type":"ContainerStarted","Data":"b5228488729828cd5618f65ecfb2361fea4de453779013bff9ea2ee15eb0ebf2"} Jul 29 14:11:52.985289 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:11:52.985229 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" podStartSLOduration=1.574332349 podStartE2EDuration="1.98521005s" totalImagesPullingTime="410.877701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:11:52.982455955 +0000 UTC m=+638.635049789" watchObservedRunningTime="2026-07-29 14:11:52.98521005 +0000 UTC m=+638.637803883" Jul 29 14:12:02.062720 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:12:02.062672 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:12:02.063150 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:12:02.062741 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:12:02.075537 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:12:02.075503 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:12:03.024662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:12:03.024635 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4" Jul 29 14:14:04.885802 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:04.885768 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d87754bdc-s6xnv"] Jul 29 14:14:04.886292 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:04.885961 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d87754bdc-s6xnv" podUID="5a9362c5-046d-4f83-941e-be629219253a" containerName="manager" containerID="cri-o://fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca" gracePeriod=30 Jul 29 14:14:05.140407 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.140350 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:14:05.192064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.192038 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a9362c5-046d-4f83-941e-be629219253a-webhook-cert\") pod \"5a9362c5-046d-4f83-941e-be629219253a\" (UID: \"5a9362c5-046d-4f83-941e-be629219253a\") " Jul 29 14:14:05.192223 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.192131 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7xpl\" (UniqueName: \"kubernetes.io/projected/5a9362c5-046d-4f83-941e-be629219253a-kube-api-access-q7xpl\") pod \"5a9362c5-046d-4f83-941e-be629219253a\" (UID: \"5a9362c5-046d-4f83-941e-be629219253a\") " Jul 29 14:14:05.193961 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.193931 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9362c5-046d-4f83-941e-be629219253a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5a9362c5-046d-4f83-941e-be629219253a" (UID: "5a9362c5-046d-4f83-941e-be629219253a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:14:05.194086 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.194060 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9362c5-046d-4f83-941e-be629219253a-kube-api-access-q7xpl" (OuterVolumeSpecName: "kube-api-access-q7xpl") pod "5a9362c5-046d-4f83-941e-be629219253a" (UID: "5a9362c5-046d-4f83-941e-be629219253a"). InnerVolumeSpecName "kube-api-access-q7xpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:14:05.293002 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.292978 2549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a9362c5-046d-4f83-941e-be629219253a-webhook-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:14:05.293002 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.292999 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7xpl\" (UniqueName: \"kubernetes.io/projected/5a9362c5-046d-4f83-941e-be629219253a-kube-api-access-q7xpl\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:14:05.590075 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.590042 2549 generic.go:356] "Generic (PLEG): container finished" podID="5a9362c5-046d-4f83-941e-be629219253a" containerID="fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca" exitCode=0 Jul 29 14:14:05.590212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.590094 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-s6xnv" Jul 29 14:14:05.590212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.590137 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-s6xnv" event={"ID":"5a9362c5-046d-4f83-941e-be629219253a","Type":"ContainerDied","Data":"fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca"} Jul 29 14:14:05.590298 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.590226 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-s6xnv" event={"ID":"5a9362c5-046d-4f83-941e-be629219253a","Type":"ContainerDied","Data":"a96451f799142ee4424ae88249548893d85636624b3cdaa6aa707bd40e096c7d"} Jul 29 14:14:05.590298 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.590255 2549 scope.go:117] "RemoveContainer" containerID="fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca" Jul 29 14:14:05.600488 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.600468 2549 scope.go:117] "RemoveContainer" containerID="fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca" Jul 29 14:14:05.600824 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:14:05.600799 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca\": container with ID starting with fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca not found: ID does not exist" containerID="fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca" Jul 29 14:14:05.600891 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.600832 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca"} err="failed to get container status \"fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca\": rpc error: code = NotFound desc = could not find container \"fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca\": container with ID starting with fb36de71728c990d6d3a8eb0f1eeb302a4e2fccdc2ad5d644ad118db5a557fca not found: ID does not exist" Jul 29 14:14:05.610501 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.610473 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d87754bdc-s6xnv"] Jul 29 14:14:05.613463 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:05.613436 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d87754bdc-s6xnv"] Jul 29 14:14:06.864121 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:14:06.864070 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a9362c5-046d-4f83-941e-be629219253a" path="/var/lib/kubelet/pods/5a9362c5-046d-4f83-941e-be629219253a/volumes" Jul 29 14:15:06.449831 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.449794 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d87754bdc-8jkr2"] Jul 29 14:15:06.450265 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.450173 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9362c5-046d-4f83-941e-be629219253a" containerName="manager" Jul 29 14:15:06.450265 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.450185 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9362c5-046d-4f83-941e-be629219253a" containerName="manager" Jul 29 14:15:06.450265 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.450257 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a9362c5-046d-4f83-941e-be629219253a" containerName="manager" Jul 29 14:15:06.452822 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.452802 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:06.455754 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.455722 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 14:15:06.456816 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.456780 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 14:15:06.456816 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.456813 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c8bls\"" Jul 29 14:15:06.462089 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.462063 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d87754bdc-8jkr2"] Jul 29 14:15:06.534054 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.534013 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24jb\" (UniqueName: \"kubernetes.io/projected/559d5738-21f9-42fc-af37-1de0853380c8-kube-api-access-x24jb\") pod \"maas-controller-d87754bdc-8jkr2\" (UID: \"559d5738-21f9-42fc-af37-1de0853380c8\") " pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:06.534054 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.534054 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559d5738-21f9-42fc-af37-1de0853380c8-webhook-cert\") pod \"maas-controller-d87754bdc-8jkr2\" (UID: \"559d5738-21f9-42fc-af37-1de0853380c8\") " pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:06.634674 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.634639 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24jb\" (UniqueName: \"kubernetes.io/projected/559d5738-21f9-42fc-af37-1de0853380c8-kube-api-access-x24jb\") pod \"maas-controller-d87754bdc-8jkr2\" (UID: \"559d5738-21f9-42fc-af37-1de0853380c8\") " pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:06.634674 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.634677 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559d5738-21f9-42fc-af37-1de0853380c8-webhook-cert\") pod \"maas-controller-d87754bdc-8jkr2\" (UID: \"559d5738-21f9-42fc-af37-1de0853380c8\") " pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:06.637176 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.637155 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559d5738-21f9-42fc-af37-1de0853380c8-webhook-cert\") pod \"maas-controller-d87754bdc-8jkr2\" (UID: \"559d5738-21f9-42fc-af37-1de0853380c8\") " pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:06.642874 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.642848 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24jb\" (UniqueName: \"kubernetes.io/projected/559d5738-21f9-42fc-af37-1de0853380c8-kube-api-access-x24jb\") pod \"maas-controller-d87754bdc-8jkr2\" (UID: \"559d5738-21f9-42fc-af37-1de0853380c8\") " pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:06.764878 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.764836 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:06.908493 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:06.908365 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d87754bdc-8jkr2"] Jul 29 14:15:06.911451 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:15:06.911418 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559d5738_21f9_42fc_af37_1de0853380c8.slice/crio-02e0d0688a32cc3d61db52f8177af11f138372d1951e9aa1a352d2852b37d057 WatchSource:0}: Error finding container 02e0d0688a32cc3d61db52f8177af11f138372d1951e9aa1a352d2852b37d057: Status 404 returned error can't find the container with id 02e0d0688a32cc3d61db52f8177af11f138372d1951e9aa1a352d2852b37d057 Jul 29 14:15:07.885821 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:07.885782 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-8jkr2" event={"ID":"559d5738-21f9-42fc-af37-1de0853380c8","Type":"ContainerStarted","Data":"02e0d0688a32cc3d61db52f8177af11f138372d1951e9aa1a352d2852b37d057"} Jul 29 14:15:08.891541 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:08.891489 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-8jkr2" event={"ID":"559d5738-21f9-42fc-af37-1de0853380c8","Type":"ContainerStarted","Data":"7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900"} Jul 29 14:15:08.892049 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:08.891705 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:08.910826 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:08.910774 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d87754bdc-8jkr2" podStartSLOduration=1.15465768 podStartE2EDuration="2.91075698s" totalImagesPullingTime="1.7560993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:15:08.909562446 +0000 UTC m=+834.562156280" watchObservedRunningTime="2026-07-29 14:15:08.91075698 +0000 UTC m=+834.563350812" Jul 29 14:15:19.899385 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:19.899351 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:24.572580 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.572539 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d87754bdc-8jkr2"] Jul 29 14:15:24.573166 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.572756 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d87754bdc-8jkr2" podUID="559d5738-21f9-42fc-af37-1de0853380c8" containerName="manager" containerID="cri-o://7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900" gracePeriod=30 Jul 29 14:15:24.833031 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.832965 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:24.895745 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.895705 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559d5738-21f9-42fc-af37-1de0853380c8-webhook-cert\") pod \"559d5738-21f9-42fc-af37-1de0853380c8\" (UID: \"559d5738-21f9-42fc-af37-1de0853380c8\") " Jul 29 14:15:24.895924 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.895766 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x24jb\" (UniqueName: \"kubernetes.io/projected/559d5738-21f9-42fc-af37-1de0853380c8-kube-api-access-x24jb\") pod \"559d5738-21f9-42fc-af37-1de0853380c8\" (UID: \"559d5738-21f9-42fc-af37-1de0853380c8\") " Jul 29 14:15:24.897999 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.897964 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559d5738-21f9-42fc-af37-1de0853380c8-kube-api-access-x24jb" (OuterVolumeSpecName: "kube-api-access-x24jb") pod "559d5738-21f9-42fc-af37-1de0853380c8" (UID: "559d5738-21f9-42fc-af37-1de0853380c8"). InnerVolumeSpecName "kube-api-access-x24jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:15:24.897999 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.897989 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559d5738-21f9-42fc-af37-1de0853380c8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "559d5738-21f9-42fc-af37-1de0853380c8" (UID: "559d5738-21f9-42fc-af37-1de0853380c8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:15:24.966266 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.966227 2549 generic.go:356] "Generic (PLEG): container finished" podID="559d5738-21f9-42fc-af37-1de0853380c8" containerID="7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900" exitCode=0 Jul 29 14:15:24.966487 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.966278 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-8jkr2" Jul 29 14:15:24.966487 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.966348 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-8jkr2" event={"ID":"559d5738-21f9-42fc-af37-1de0853380c8","Type":"ContainerDied","Data":"7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900"} Jul 29 14:15:24.966487 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.966417 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-8jkr2" event={"ID":"559d5738-21f9-42fc-af37-1de0853380c8","Type":"ContainerDied","Data":"02e0d0688a32cc3d61db52f8177af11f138372d1951e9aa1a352d2852b37d057"} Jul 29 14:15:24.966487 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.966448 2549 scope.go:117] "RemoveContainer" containerID="7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900" Jul 29 14:15:24.976637 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.976617 2549 scope.go:117] "RemoveContainer" containerID="7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900" Jul 29 14:15:24.976912 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:15:24.976893 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900\": container with ID starting with 7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900 not found: ID does not exist" containerID="7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900" Jul 29 14:15:24.976962 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.976921 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900"} err="failed to get container status \"7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900\": rpc error: code = NotFound desc = could not find container \"7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900\": container with ID starting with 7ed7edfcaece17272573c6f5de9bee093dd3be349645bc960453d4b2b3844900 not found: ID does not exist" Jul 29 14:15:24.990004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.989969 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d87754bdc-8jkr2"] Jul 29 14:15:24.991641 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.991618 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d87754bdc-8jkr2"] Jul 29 14:15:24.997000 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.996976 2549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/559d5738-21f9-42fc-af37-1de0853380c8-webhook-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:15:24.997098 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:24.997004 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x24jb\" (UniqueName: \"kubernetes.io/projected/559d5738-21f9-42fc-af37-1de0853380c8-kube-api-access-x24jb\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:15:25.812498 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.812458 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d87754bdc-lxjx5"] Jul 29 14:15:25.813037 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.812987 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="559d5738-21f9-42fc-af37-1de0853380c8" containerName="manager" Jul 29 14:15:25.813037 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.813008 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="559d5738-21f9-42fc-af37-1de0853380c8" containerName="manager" Jul 29 14:15:25.813164 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.813118 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="559d5738-21f9-42fc-af37-1de0853380c8" containerName="manager" Jul 29 14:15:25.817485 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.817459 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:15:25.821105 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.821071 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c8bls\"" Jul 29 14:15:25.821105 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.821096 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 14:15:25.821105 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.821071 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 14:15:25.828426 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.828399 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d87754bdc-lxjx5"] Jul 29 14:15:25.905196 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.905161 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574cd\" (UniqueName: \"kubernetes.io/projected/1664efef-af3b-432c-a26c-5c16db455cb8-kube-api-access-574cd\") pod \"maas-controller-d87754bdc-lxjx5\" (UID: \"1664efef-af3b-432c-a26c-5c16db455cb8\") " pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:15:25.905444 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:25.905236 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1664efef-af3b-432c-a26c-5c16db455cb8-webhook-cert\") pod \"maas-controller-d87754bdc-lxjx5\" (UID: \"1664efef-af3b-432c-a26c-5c16db455cb8\") " pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:15:26.005744 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:26.005699 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574cd\" (UniqueName: \"kubernetes.io/projected/1664efef-af3b-432c-a26c-5c16db455cb8-kube-api-access-574cd\") pod \"maas-controller-d87754bdc-lxjx5\" (UID: \"1664efef-af3b-432c-a26c-5c16db455cb8\") " pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:15:26.005979 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:26.005790 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1664efef-af3b-432c-a26c-5c16db455cb8-webhook-cert\") pod \"maas-controller-d87754bdc-lxjx5\" (UID: \"1664efef-af3b-432c-a26c-5c16db455cb8\") " pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:15:26.008486 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:26.008462 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1664efef-af3b-432c-a26c-5c16db455cb8-webhook-cert\") pod \"maas-controller-d87754bdc-lxjx5\" (UID: \"1664efef-af3b-432c-a26c-5c16db455cb8\") " pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:15:26.013571 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:26.013544 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-574cd\" (UniqueName: \"kubernetes.io/projected/1664efef-af3b-432c-a26c-5c16db455cb8-kube-api-access-574cd\") pod \"maas-controller-d87754bdc-lxjx5\" (UID: \"1664efef-af3b-432c-a26c-5c16db455cb8\") " pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:15:26.128391 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:26.128289 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:15:26.267381 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:26.267297 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d87754bdc-lxjx5"] Jul 29 14:15:26.269834 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:15:26.269805 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1664efef_af3b_432c_a26c_5c16db455cb8.slice/crio-ae4f3825bdb75a54c9db6a529b33b997df0704ec44a402146e573a22a8825be9 WatchSource:0}: Error finding container ae4f3825bdb75a54c9db6a529b33b997df0704ec44a402146e573a22a8825be9: Status 404 returned error can't find the container with id ae4f3825bdb75a54c9db6a529b33b997df0704ec44a402146e573a22a8825be9 Jul 29 14:15:26.864920 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:26.864886 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="559d5738-21f9-42fc-af37-1de0853380c8" path="/var/lib/kubelet/pods/559d5738-21f9-42fc-af37-1de0853380c8/volumes" Jul 29 14:15:26.978198 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:26.978162 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-lxjx5" event={"ID":"1664efef-af3b-432c-a26c-5c16db455cb8","Type":"ContainerStarted","Data":"ae4f3825bdb75a54c9db6a529b33b997df0704ec44a402146e573a22a8825be9"} Jul 29 14:15:27.984432 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:27.984390 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d87754bdc-lxjx5" event={"ID":"1664efef-af3b-432c-a26c-5c16db455cb8","Type":"ContainerStarted","Data":"ed26d7e47e54152694a254342af9278547fb156b5c48110d9d17fe3e5285c0e4"} Jul 29 14:15:27.984877 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:27.984485 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:15:28.002507 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:28.002448 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d87754bdc-lxjx5" podStartSLOduration=2.012062158 podStartE2EDuration="3.002428906s" totalImagesPullingTime="990.366748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:15:28.001856501 +0000 UTC m=+853.654450355" watchObservedRunningTime="2026-07-29 14:15:28.002428906 +0000 UTC m=+853.655022739" Jul 29 14:15:38.991951 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:15:38.991910 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d87754bdc-lxjx5" Jul 29 14:16:14.804614 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:16:14.804527 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:16:14.805747 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:16:14.805725 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:21:14.833561 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:21:14.833528 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:21:14.834067 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:21:14.833564 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:26:14.862287 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:26:14.862251 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:26:14.863995 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:26:14.863968 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:28:59.377270 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.377230 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg"] Jul 29 14:28:59.380762 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.380736 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:28:59.384831 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.384804 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d8b7adbd\"" Jul 29 14:28:59.395244 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.395213 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg"] Jul 29 14:28:59.399034 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.398993 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7"] Jul 29 14:28:59.402525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.402503 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:28:59.405428 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.405386 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d8b7adbd-dockercfg-z8nsr\"" Jul 29 14:28:59.420698 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.420656 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7"] Jul 29 14:28:59.491619 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.491581 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-plugins-config-volume\") pod \"payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7\" (UID: \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\") " pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:28:59.491826 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.491706 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjf8\" (UniqueName: \"kubernetes.io/projected/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-kube-api-access-zsjf8\") pod \"payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7\" (UID: \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\") " pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:28:59.491826 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.491773 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10fae96-8c3e-4c2f-b94c-bc6687872891-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg\" (UID: \"e10fae96-8c3e-4c2f-b94c-bc6687872891\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:28:59.491826 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.491806 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25wg2\" (UniqueName: \"kubernetes.io/projected/e10fae96-8c3e-4c2f-b94c-bc6687872891-kube-api-access-25wg2\") pod \"payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg\" (UID: \"e10fae96-8c3e-4c2f-b94c-bc6687872891\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:28:59.592872 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.592824 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjf8\" (UniqueName: \"kubernetes.io/projected/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-kube-api-access-zsjf8\") pod \"payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7\" (UID: \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\") " pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:28:59.592872 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.592880 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10fae96-8c3e-4c2f-b94c-bc6687872891-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg\" (UID: \"e10fae96-8c3e-4c2f-b94c-bc6687872891\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:28:59.593156 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.592906 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25wg2\" (UniqueName: \"kubernetes.io/projected/e10fae96-8c3e-4c2f-b94c-bc6687872891-kube-api-access-25wg2\") pod \"payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg\" (UID: \"e10fae96-8c3e-4c2f-b94c-bc6687872891\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:28:59.593156 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.592945 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-plugins-config-volume\") pod \"payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7\" (UID: \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\") " pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:28:59.593661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.593637 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-plugins-config-volume\") pod \"payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7\" (UID: \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\") " pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:28:59.593775 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.593733 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10fae96-8c3e-4c2f-b94c-bc6687872891-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg\" (UID: \"e10fae96-8c3e-4c2f-b94c-bc6687872891\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:28:59.603331 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.603274 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25wg2\" (UniqueName: \"kubernetes.io/projected/e10fae96-8c3e-4c2f-b94c-bc6687872891-kube-api-access-25wg2\") pod \"payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg\" (UID: \"e10fae96-8c3e-4c2f-b94c-bc6687872891\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:28:59.604380 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.604356 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjf8\" (UniqueName: \"kubernetes.io/projected/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-kube-api-access-zsjf8\") pod \"payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7\" (UID: \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\") " pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:28:59.691683 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.691581 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:28:59.714277 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.714161 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:28:59.855437 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.855401 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg"] Jul 29 14:28:59.860142 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:28:59.860112 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10fae96_8c3e_4c2f_b94c_bc6687872891.slice/crio-d5587f25ade701a696bd0b5a879fc1b0be104dd8c1f6fdbc4f7199a934fd6f46 WatchSource:0}: Error finding container d5587f25ade701a696bd0b5a879fc1b0be104dd8c1f6fdbc4f7199a934fd6f46: Status 404 returned error can't find the container with id d5587f25ade701a696bd0b5a879fc1b0be104dd8c1f6fdbc4f7199a934fd6f46 Jul 29 14:28:59.890951 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:28:59.888557 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7"] Jul 29 14:28:59.895811 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:28:59.895773 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4a0f0b_302f_4cd2_966a_9a8e691012a2.slice/crio-c2552e0ce7abfdf19335f239465c554af3d95e3e2a62c2341abded5ac5a89ed7 WatchSource:0}: Error finding container c2552e0ce7abfdf19335f239465c554af3d95e3e2a62c2341abded5ac5a89ed7: Status 404 returned error can't find the container with id c2552e0ce7abfdf19335f239465c554af3d95e3e2a62c2341abded5ac5a89ed7 Jul 29 14:29:00.830153 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:00.830110 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" event={"ID":"e10fae96-8c3e-4c2f-b94c-bc6687872891","Type":"ContainerStarted","Data":"d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6"} Jul 29 14:29:00.830712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:00.830184 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" event={"ID":"e10fae96-8c3e-4c2f-b94c-bc6687872891","Type":"ContainerStarted","Data":"d5587f25ade701a696bd0b5a879fc1b0be104dd8c1f6fdbc4f7199a934fd6f46"} Jul 29 14:29:00.830712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:00.830246 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:29:00.831820 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:00.831793 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" event={"ID":"fe4a0f0b-302f-4cd2-966a-9a8e691012a2","Type":"ContainerStarted","Data":"267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c"} Jul 29 14:29:00.831933 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:00.831860 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:29:00.831933 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:00.831878 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" event={"ID":"fe4a0f0b-302f-4cd2-966a-9a8e691012a2","Type":"ContainerStarted","Data":"c2552e0ce7abfdf19335f239465c554af3d95e3e2a62c2341abded5ac5a89ed7"} Jul 29 14:29:00.849756 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:00.849688 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" podStartSLOduration=1.849665846 podStartE2EDuration="1.849665846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:29:00.847413744 +0000 UTC m=+1666.500007578" watchObservedRunningTime="2026-07-29 14:29:00.849665846 +0000 UTC m=+1666.502259684" Jul 29 14:29:00.867230 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:00.867181 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" podStartSLOduration=1.8671658880000002 podStartE2EDuration="1.867165888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:29:00.865176003 +0000 UTC m=+1666.517769838" watchObservedRunningTime="2026-07-29 14:29:00.867165888 +0000 UTC m=+1666.519759755" Jul 29 14:29:11.838699 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:11.838663 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:29:11.839189 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:11.838754 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:29:29.203801 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.203760 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7"] Jul 29 14:29:29.206936 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.204122 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" podUID="fe4a0f0b-302f-4cd2-966a-9a8e691012a2" containerName="payload-processing" containerID="cri-o://267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c" gracePeriod=30 Jul 29 14:29:29.212393 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.212229 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg"] Jul 29 14:29:29.212931 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.212519 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" podUID="e10fae96-8c3e-4c2f-b94c-bc6687872891" containerName="payload-pre-processing" containerID="cri-o://d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6" gracePeriod=30 Jul 29 14:29:29.484658 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.484626 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:29:29.519175 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.519141 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:29:29.593436 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.593390 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10fae96-8c3e-4c2f-b94c-bc6687872891-plugins-config-volume\") pod \"e10fae96-8c3e-4c2f-b94c-bc6687872891\" (UID: \"e10fae96-8c3e-4c2f-b94c-bc6687872891\") " Jul 29 14:29:29.593436 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.593441 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-plugins-config-volume\") pod \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\" (UID: \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\") " Jul 29 14:29:29.593720 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.593493 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25wg2\" (UniqueName: \"kubernetes.io/projected/e10fae96-8c3e-4c2f-b94c-bc6687872891-kube-api-access-25wg2\") pod \"e10fae96-8c3e-4c2f-b94c-bc6687872891\" (UID: \"e10fae96-8c3e-4c2f-b94c-bc6687872891\") " Jul 29 14:29:29.593720 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.593545 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsjf8\" (UniqueName: \"kubernetes.io/projected/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-kube-api-access-zsjf8\") pod \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\" (UID: \"fe4a0f0b-302f-4cd2-966a-9a8e691012a2\") " Jul 29 14:29:29.593907 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.593879 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10fae96-8c3e-4c2f-b94c-bc6687872891-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e10fae96-8c3e-4c2f-b94c-bc6687872891" (UID: "e10fae96-8c3e-4c2f-b94c-bc6687872891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:29:29.593978 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.593879 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe4a0f0b-302f-4cd2-966a-9a8e691012a2" (UID: "fe4a0f0b-302f-4cd2-966a-9a8e691012a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:29:29.595957 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.595928 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10fae96-8c3e-4c2f-b94c-bc6687872891-kube-api-access-25wg2" (OuterVolumeSpecName: "kube-api-access-25wg2") pod "e10fae96-8c3e-4c2f-b94c-bc6687872891" (UID: "e10fae96-8c3e-4c2f-b94c-bc6687872891"). InnerVolumeSpecName "kube-api-access-25wg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:29:29.596013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.595968 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-kube-api-access-zsjf8" (OuterVolumeSpecName: "kube-api-access-zsjf8") pod "fe4a0f0b-302f-4cd2-966a-9a8e691012a2" (UID: "fe4a0f0b-302f-4cd2-966a-9a8e691012a2"). InnerVolumeSpecName "kube-api-access-zsjf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:29:29.694997 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.694958 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10fae96-8c3e-4c2f-b94c-bc6687872891-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:29:29.694997 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.694988 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:29:29.694997 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.694998 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25wg2\" (UniqueName: \"kubernetes.io/projected/e10fae96-8c3e-4c2f-b94c-bc6687872891-kube-api-access-25wg2\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:29:29.694997 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.695009 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsjf8\" (UniqueName: \"kubernetes.io/projected/fe4a0f0b-302f-4cd2-966a-9a8e691012a2-kube-api-access-zsjf8\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:29:29.994267 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.994233 2549 generic.go:356] "Generic (PLEG): container finished" podID="fe4a0f0b-302f-4cd2-966a-9a8e691012a2" containerID="267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c" exitCode=0 Jul 29 14:29:29.994522 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.994280 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" Jul 29 14:29:29.994522 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.994327 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" event={"ID":"fe4a0f0b-302f-4cd2-966a-9a8e691012a2","Type":"ContainerDied","Data":"267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c"} Jul 29 14:29:29.994522 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.994391 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7" event={"ID":"fe4a0f0b-302f-4cd2-966a-9a8e691012a2","Type":"ContainerDied","Data":"c2552e0ce7abfdf19335f239465c554af3d95e3e2a62c2341abded5ac5a89ed7"} Jul 29 14:29:29.994522 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.994422 2549 scope.go:117] "RemoveContainer" containerID="267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c" Jul 29 14:29:29.995859 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.995837 2549 generic.go:356] "Generic (PLEG): container finished" podID="e10fae96-8c3e-4c2f-b94c-bc6687872891" containerID="d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6" exitCode=0 Jul 29 14:29:29.995995 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.995887 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" Jul 29 14:29:29.995995 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.995903 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" event={"ID":"e10fae96-8c3e-4c2f-b94c-bc6687872891","Type":"ContainerDied","Data":"d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6"} Jul 29 14:29:29.995995 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:29.995960 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg" event={"ID":"e10fae96-8c3e-4c2f-b94c-bc6687872891","Type":"ContainerDied","Data":"d5587f25ade701a696bd0b5a879fc1b0be104dd8c1f6fdbc4f7199a934fd6f46"} Jul 29 14:29:30.006734 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.006672 2549 scope.go:117] "RemoveContainer" containerID="267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c" Jul 29 14:29:30.007251 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:29:30.007186 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c\": container with ID starting with 267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c not found: ID does not exist" containerID="267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c" Jul 29 14:29:30.007423 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.007276 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c"} err="failed to get container status \"267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c\": rpc error: code = NotFound desc = could not find container \"267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c\": container with ID starting with 267d44db2dc50a2d7f1c1acd4d72af45c44cb4c3ca25ea51a5881e5683b7e57c not found: ID does not exist" Jul 29 14:29:30.007423 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.007298 2549 scope.go:117] "RemoveContainer" containerID="d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6" Jul 29 14:29:30.018250 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.018223 2549 scope.go:117] "RemoveContainer" containerID="d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6" Jul 29 14:29:30.018649 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:29:30.018624 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6\": container with ID starting with d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6 not found: ID does not exist" containerID="d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6" Jul 29 14:29:30.018748 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.018658 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6"} err="failed to get container status \"d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6\": rpc error: code = NotFound desc = could not find container \"d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6\": container with ID starting with d8fc667f930f1244831a6bb7459a6fb98b9ead9e9be06a77b0be96293f6ab5a6 not found: ID does not exist" Jul 29 14:29:30.028131 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.028099 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7"] Jul 29 14:29:30.035827 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.035794 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8b7adbd-7d697769b-zsdm7"] Jul 29 14:29:30.048393 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.048360 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg"] Jul 29 14:29:30.059183 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.051396 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8b7adbd-5b7c777696-c6hmg"] Jul 29 14:29:30.865538 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.865496 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10fae96-8c3e-4c2f-b94c-bc6687872891" path="/var/lib/kubelet/pods/e10fae96-8c3e-4c2f-b94c-bc6687872891/volumes" Jul 29 14:29:30.865950 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:30.865877 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe4a0f0b-302f-4cd2-966a-9a8e691012a2" path="/var/lib/kubelet/pods/fe4a0f0b-302f-4cd2-966a-9a8e691012a2/volumes" Jul 29 14:29:33.912968 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.912923 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm"] Jul 29 14:29:33.913411 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.913306 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10fae96-8c3e-4c2f-b94c-bc6687872891" containerName="payload-pre-processing" Jul 29 14:29:33.913411 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.913331 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10fae96-8c3e-4c2f-b94c-bc6687872891" containerName="payload-pre-processing" Jul 29 14:29:33.913411 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.913359 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe4a0f0b-302f-4cd2-966a-9a8e691012a2" containerName="payload-processing" Jul 29 14:29:33.913411 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.913365 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe4a0f0b-302f-4cd2-966a-9a8e691012a2" containerName="payload-processing" Jul 29 14:29:33.913764 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.913439 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe4a0f0b-302f-4cd2-966a-9a8e691012a2" containerName="payload-processing" Jul 29 14:29:33.913764 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.913447 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10fae96-8c3e-4c2f-b94c-bc6687872891" containerName="payload-pre-processing" Jul 29 14:29:33.918008 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.917981 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:33.920845 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.920819 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-59408d2e\"" Jul 29 14:29:33.923447 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.923426 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-59408d2e-dockercfg-blh4m\"" Jul 29 14:29:33.938000 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.937903 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm"] Jul 29 14:29:33.946031 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.945958 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n"] Jul 29 14:29:33.950036 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.950006 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:29:33.961867 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:33.961833 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n"] Jul 29 14:29:34.038934 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.038892 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zsnk\" (UniqueName: \"kubernetes.io/projected/26a4efdc-d35a-407c-abf0-f46b274629ec-kube-api-access-7zsnk\") pod \"payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n\" (UID: \"26a4efdc-d35a-407c-abf0-f46b274629ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:29:34.038934 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.038939 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblz5\" (UniqueName: \"kubernetes.io/projected/f3e5353b-c26c-4040-b36d-0775520dd0b9-kube-api-access-tblz5\") pod \"payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm\" (UID: \"f3e5353b-c26c-4040-b36d-0775520dd0b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:34.039197 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.038979 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e5353b-c26c-4040-b36d-0775520dd0b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm\" (UID: \"f3e5353b-c26c-4040-b36d-0775520dd0b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:34.039197 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.039034 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a4efdc-d35a-407c-abf0-f46b274629ec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n\" (UID: \"26a4efdc-d35a-407c-abf0-f46b274629ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:29:34.140295 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.140257 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zsnk\" (UniqueName: \"kubernetes.io/projected/26a4efdc-d35a-407c-abf0-f46b274629ec-kube-api-access-7zsnk\") pod \"payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n\" (UID: \"26a4efdc-d35a-407c-abf0-f46b274629ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:29:34.140295 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.140301 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblz5\" (UniqueName: \"kubernetes.io/projected/f3e5353b-c26c-4040-b36d-0775520dd0b9-kube-api-access-tblz5\") pod \"payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm\" (UID: \"f3e5353b-c26c-4040-b36d-0775520dd0b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:34.140664 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.140370 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e5353b-c26c-4040-b36d-0775520dd0b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm\" (UID: \"f3e5353b-c26c-4040-b36d-0775520dd0b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:34.140664 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.140429 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a4efdc-d35a-407c-abf0-f46b274629ec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n\" (UID: \"26a4efdc-d35a-407c-abf0-f46b274629ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:29:34.141205 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.141183 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e5353b-c26c-4040-b36d-0775520dd0b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm\" (UID: \"f3e5353b-c26c-4040-b36d-0775520dd0b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:34.141205 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.141194 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a4efdc-d35a-407c-abf0-f46b274629ec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n\" (UID: \"26a4efdc-d35a-407c-abf0-f46b274629ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:29:34.148700 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.148671 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zsnk\" (UniqueName: \"kubernetes.io/projected/26a4efdc-d35a-407c-abf0-f46b274629ec-kube-api-access-7zsnk\") pod \"payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n\" (UID: \"26a4efdc-d35a-407c-abf0-f46b274629ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:29:34.148905 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.148881 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblz5\" (UniqueName: \"kubernetes.io/projected/f3e5353b-c26c-4040-b36d-0775520dd0b9-kube-api-access-tblz5\") pod \"payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm\" (UID: \"f3e5353b-c26c-4040-b36d-0775520dd0b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:34.228702 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.228603 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:34.262628 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.262589 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:29:34.397797 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.396027 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm"] Jul 29 14:29:34.403294 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:29:34.403250 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e5353b_c26c_4040_b36d_0775520dd0b9.slice/crio-d448e7bd12e8b4d1b809f6ab3e2cc08f1be361b85e5f28261209b3c88038694d WatchSource:0}: Error finding container d448e7bd12e8b4d1b809f6ab3e2cc08f1be361b85e5f28261209b3c88038694d: Status 404 returned error can't find the container with id d448e7bd12e8b4d1b809f6ab3e2cc08f1be361b85e5f28261209b3c88038694d Jul 29 14:29:34.437284 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:34.436615 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n"] Jul 29 14:29:34.439374 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:29:34.439279 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a4efdc_d35a_407c_abf0_f46b274629ec.slice/crio-6b4910c2957f001692ebdb5d60f8cc6852d02aeaec49062aecace959c9e95efa WatchSource:0}: Error finding container 6b4910c2957f001692ebdb5d60f8cc6852d02aeaec49062aecace959c9e95efa: Status 404 returned error can't find the container with id 6b4910c2957f001692ebdb5d60f8cc6852d02aeaec49062aecace959c9e95efa Jul 29 14:29:35.025431 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:35.025389 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" event={"ID":"26a4efdc-d35a-407c-abf0-f46b274629ec","Type":"ContainerStarted","Data":"02cf356aed60217c1954ed85b8d4755eb10c6c0c3202392d97624d1eb8a44df6"} Jul 29 14:29:35.025952 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:35.025492 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" event={"ID":"26a4efdc-d35a-407c-abf0-f46b274629ec","Type":"ContainerStarted","Data":"6b4910c2957f001692ebdb5d60f8cc6852d02aeaec49062aecace959c9e95efa"} Jul 29 14:29:35.025952 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:35.025522 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:29:35.027142 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:35.027116 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" event={"ID":"f3e5353b-c26c-4040-b36d-0775520dd0b9","Type":"ContainerStarted","Data":"10c6290367a14fdfdf68bbdc847fc9f59fd2313277b38ab17ee612a42ddc2b0b"} Jul 29 14:29:35.027263 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:35.027160 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" event={"ID":"f3e5353b-c26c-4040-b36d-0775520dd0b9","Type":"ContainerStarted","Data":"d448e7bd12e8b4d1b809f6ab3e2cc08f1be361b85e5f28261209b3c88038694d"} Jul 29 14:29:35.027263 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:35.027209 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:35.043559 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:35.043494 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" podStartSLOduration=2.043478895 podStartE2EDuration="2.043478895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:29:35.042710376 +0000 UTC m=+1700.695304214" watchObservedRunningTime="2026-07-29 14:29:35.043478895 +0000 UTC m=+1700.696072729" Jul 29 14:29:35.058766 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:35.058700 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" podStartSLOduration=2.058679673 podStartE2EDuration="2.058679673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:29:35.057917691 +0000 UTC m=+1700.710511563" watchObservedRunningTime="2026-07-29 14:29:35.058679673 +0000 UTC m=+1700.711273507" Jul 29 14:29:46.034362 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:46.034254 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:29:46.034362 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:29:46.034360 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:30:03.937960 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:03.937914 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n"] Jul 29 14:30:03.938538 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:03.938241 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" podUID="26a4efdc-d35a-407c-abf0-f46b274629ec" containerName="payload-processing" containerID="cri-o://02cf356aed60217c1954ed85b8d4755eb10c6c0c3202392d97624d1eb8a44df6" gracePeriod=30 Jul 29 14:30:03.947829 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:03.947790 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm"] Jul 29 14:30:03.948338 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:03.948278 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" podUID="f3e5353b-c26c-4040-b36d-0775520dd0b9" containerName="payload-pre-processing" containerID="cri-o://10c6290367a14fdfdf68bbdc847fc9f59fd2313277b38ab17ee612a42ddc2b0b" gracePeriod=30 Jul 29 14:30:04.178406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.178360 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" event={"ID":"26a4efdc-d35a-407c-abf0-f46b274629ec","Type":"ContainerDied","Data":"02cf356aed60217c1954ed85b8d4755eb10c6c0c3202392d97624d1eb8a44df6"} Jul 29 14:30:04.178615 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.178363 2549 generic.go:356] "Generic (PLEG): container finished" podID="26a4efdc-d35a-407c-abf0-f46b274629ec" containerID="02cf356aed60217c1954ed85b8d4755eb10c6c0c3202392d97624d1eb8a44df6" exitCode=0 Jul 29 14:30:04.180450 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.180411 2549 generic.go:356] "Generic (PLEG): container finished" podID="f3e5353b-c26c-4040-b36d-0775520dd0b9" containerID="10c6290367a14fdfdf68bbdc847fc9f59fd2313277b38ab17ee612a42ddc2b0b" exitCode=0 Jul 29 14:30:04.180611 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.180576 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" event={"ID":"f3e5353b-c26c-4040-b36d-0775520dd0b9","Type":"ContainerDied","Data":"10c6290367a14fdfdf68bbdc847fc9f59fd2313277b38ab17ee612a42ddc2b0b"} Jul 29 14:30:04.255859 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.255824 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:30:04.259153 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.259124 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:30:04.325749 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.325711 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tblz5\" (UniqueName: \"kubernetes.io/projected/f3e5353b-c26c-4040-b36d-0775520dd0b9-kube-api-access-tblz5\") pod \"f3e5353b-c26c-4040-b36d-0775520dd0b9\" (UID: \"f3e5353b-c26c-4040-b36d-0775520dd0b9\") " Jul 29 14:30:04.325964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.325829 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zsnk\" (UniqueName: \"kubernetes.io/projected/26a4efdc-d35a-407c-abf0-f46b274629ec-kube-api-access-7zsnk\") pod \"26a4efdc-d35a-407c-abf0-f46b274629ec\" (UID: \"26a4efdc-d35a-407c-abf0-f46b274629ec\") " Jul 29 14:30:04.325964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.325878 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e5353b-c26c-4040-b36d-0775520dd0b9-plugins-config-volume\") pod \"f3e5353b-c26c-4040-b36d-0775520dd0b9\" (UID: \"f3e5353b-c26c-4040-b36d-0775520dd0b9\") " Jul 29 14:30:04.325964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.325901 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a4efdc-d35a-407c-abf0-f46b274629ec-plugins-config-volume\") pod \"26a4efdc-d35a-407c-abf0-f46b274629ec\" (UID: \"26a4efdc-d35a-407c-abf0-f46b274629ec\") " Jul 29 14:30:04.326358 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.326304 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e5353b-c26c-4040-b36d-0775520dd0b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3e5353b-c26c-4040-b36d-0775520dd0b9" (UID: "f3e5353b-c26c-4040-b36d-0775520dd0b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:30:04.326358 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.326343 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a4efdc-d35a-407c-abf0-f46b274629ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26a4efdc-d35a-407c-abf0-f46b274629ec" (UID: "26a4efdc-d35a-407c-abf0-f46b274629ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:30:04.328237 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.328204 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a4efdc-d35a-407c-abf0-f46b274629ec-kube-api-access-7zsnk" (OuterVolumeSpecName: "kube-api-access-7zsnk") pod "26a4efdc-d35a-407c-abf0-f46b274629ec" (UID: "26a4efdc-d35a-407c-abf0-f46b274629ec"). InnerVolumeSpecName "kube-api-access-7zsnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:30:04.328393 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.328244 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e5353b-c26c-4040-b36d-0775520dd0b9-kube-api-access-tblz5" (OuterVolumeSpecName: "kube-api-access-tblz5") pod "f3e5353b-c26c-4040-b36d-0775520dd0b9" (UID: "f3e5353b-c26c-4040-b36d-0775520dd0b9"). InnerVolumeSpecName "kube-api-access-tblz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:30:04.426928 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.426880 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zsnk\" (UniqueName: \"kubernetes.io/projected/26a4efdc-d35a-407c-abf0-f46b274629ec-kube-api-access-7zsnk\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:30:04.426928 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.426916 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e5353b-c26c-4040-b36d-0775520dd0b9-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:30:04.426928 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.426926 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a4efdc-d35a-407c-abf0-f46b274629ec-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:30:04.426928 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:04.426935 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tblz5\" (UniqueName: \"kubernetes.io/projected/f3e5353b-c26c-4040-b36d-0775520dd0b9-kube-api-access-tblz5\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:30:05.188184 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.188075 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" event={"ID":"26a4efdc-d35a-407c-abf0-f46b274629ec","Type":"ContainerDied","Data":"6b4910c2957f001692ebdb5d60f8cc6852d02aeaec49062aecace959c9e95efa"} Jul 29 14:30:05.188184 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.188154 2549 scope.go:117] "RemoveContainer" containerID="02cf356aed60217c1954ed85b8d4755eb10c6c0c3202392d97624d1eb8a44df6" Jul 29 14:30:05.188184 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.188084 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n" Jul 29 14:30:05.189636 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.189614 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" Jul 29 14:30:05.189728 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.189605 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm" event={"ID":"f3e5353b-c26c-4040-b36d-0775520dd0b9","Type":"ContainerDied","Data":"d448e7bd12e8b4d1b809f6ab3e2cc08f1be361b85e5f28261209b3c88038694d"} Jul 29 14:30:05.197591 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.197570 2549 scope.go:117] "RemoveContainer" containerID="10c6290367a14fdfdf68bbdc847fc9f59fd2313277b38ab17ee612a42ddc2b0b" Jul 29 14:30:05.210287 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.210254 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n"] Jul 29 14:30:05.218982 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.218948 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59408d2e-c78fc6c69-cf99n"] Jul 29 14:30:05.230124 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.230089 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm"] Jul 29 14:30:05.236457 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:05.236418 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59408d2e-644b9d4c56-xk8jm"] Jul 29 14:30:06.864576 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:06.864540 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a4efdc-d35a-407c-abf0-f46b274629ec" path="/var/lib/kubelet/pods/26a4efdc-d35a-407c-abf0-f46b274629ec/volumes" Jul 29 14:30:06.864988 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:06.864936 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3e5353b-c26c-4040-b36d-0775520dd0b9" path="/var/lib/kubelet/pods/f3e5353b-c26c-4040-b36d-0775520dd0b9/volumes" Jul 29 14:30:08.824848 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.824811 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj"] Jul 29 14:30:08.825382 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.825230 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a4efdc-d35a-407c-abf0-f46b274629ec" containerName="payload-processing" Jul 29 14:30:08.825382 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.825243 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a4efdc-d35a-407c-abf0-f46b274629ec" containerName="payload-processing" Jul 29 14:30:08.825382 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.825275 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3e5353b-c26c-4040-b36d-0775520dd0b9" containerName="payload-pre-processing" Jul 29 14:30:08.825382 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.825281 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3e5353b-c26c-4040-b36d-0775520dd0b9" containerName="payload-pre-processing" Jul 29 14:30:08.832220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.825947 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a4efdc-d35a-407c-abf0-f46b274629ec" containerName="payload-processing" Jul 29 14:30:08.832220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.826007 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3e5353b-c26c-4040-b36d-0775520dd0b9" containerName="payload-pre-processing" Jul 29 14:30:08.832220 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.832075 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:08.836240 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.836201 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ee08ac7d\"" Jul 29 14:30:08.836399 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.836340 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ee08ac7d-dockercfg-md4tc\"" Jul 29 14:30:08.847613 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.847582 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj"] Jul 29 14:30:08.871679 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.871630 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc4vn\" (UniqueName: \"kubernetes.io/projected/69222735-bb93-4e8d-ad2f-a67133cc8325-kube-api-access-vc4vn\") pod \"payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj\" (UID: \"69222735-bb93-4e8d-ad2f-a67133cc8325\") " pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:08.871900 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.871692 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69222735-bb93-4e8d-ad2f-a67133cc8325-plugins-config-volume\") pod \"payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj\" (UID: \"69222735-bb93-4e8d-ad2f-a67133cc8325\") " pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:08.876987 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.876948 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt"] Jul 29 14:30:08.880485 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.880462 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:08.897814 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.897778 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt"] Jul 29 14:30:08.973058 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.973016 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljlkd\" (UniqueName: \"kubernetes.io/projected/fb22f637-9157-4c15-8706-2c67001f6283-kube-api-access-ljlkd\") pod \"payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt\" (UID: \"fb22f637-9157-4c15-8706-2c67001f6283\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:08.973287 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.973069 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb22f637-9157-4c15-8706-2c67001f6283-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt\" (UID: \"fb22f637-9157-4c15-8706-2c67001f6283\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:08.973287 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.973223 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc4vn\" (UniqueName: \"kubernetes.io/projected/69222735-bb93-4e8d-ad2f-a67133cc8325-kube-api-access-vc4vn\") pod \"payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj\" (UID: \"69222735-bb93-4e8d-ad2f-a67133cc8325\") " pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:08.973448 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.973295 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69222735-bb93-4e8d-ad2f-a67133cc8325-plugins-config-volume\") pod \"payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj\" (UID: \"69222735-bb93-4e8d-ad2f-a67133cc8325\") " pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:08.973951 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.973932 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69222735-bb93-4e8d-ad2f-a67133cc8325-plugins-config-volume\") pod \"payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj\" (UID: \"69222735-bb93-4e8d-ad2f-a67133cc8325\") " pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:08.984434 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:08.984401 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc4vn\" (UniqueName: \"kubernetes.io/projected/69222735-bb93-4e8d-ad2f-a67133cc8325-kube-api-access-vc4vn\") pod \"payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj\" (UID: \"69222735-bb93-4e8d-ad2f-a67133cc8325\") " pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:09.074794 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:09.074757 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljlkd\" (UniqueName: \"kubernetes.io/projected/fb22f637-9157-4c15-8706-2c67001f6283-kube-api-access-ljlkd\") pod \"payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt\" (UID: \"fb22f637-9157-4c15-8706-2c67001f6283\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:09.075031 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:09.074856 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb22f637-9157-4c15-8706-2c67001f6283-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt\" (UID: \"fb22f637-9157-4c15-8706-2c67001f6283\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:09.075610 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:09.075590 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb22f637-9157-4c15-8706-2c67001f6283-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt\" (UID: \"fb22f637-9157-4c15-8706-2c67001f6283\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:09.083721 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:09.083689 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljlkd\" (UniqueName: \"kubernetes.io/projected/fb22f637-9157-4c15-8706-2c67001f6283-kube-api-access-ljlkd\") pod \"payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt\" (UID: \"fb22f637-9157-4c15-8706-2c67001f6283\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:09.143914 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:09.143875 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:09.191196 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:09.191158 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:09.312848 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:09.312815 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj"] Jul 29 14:30:09.316575 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:30:09.316544 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69222735_bb93_4e8d_ad2f_a67133cc8325.slice/crio-c727d32e1cb4bcd2167cdfeaa3e8de905dc33d2a1fe232b1606bec37abaa34b3 WatchSource:0}: Error finding container c727d32e1cb4bcd2167cdfeaa3e8de905dc33d2a1fe232b1606bec37abaa34b3: Status 404 returned error can't find the container with id c727d32e1cb4bcd2167cdfeaa3e8de905dc33d2a1fe232b1606bec37abaa34b3 Jul 29 14:30:09.365113 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:09.365082 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt"] Jul 29 14:30:09.368630 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:30:09.368593 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb22f637_9157_4c15_8706_2c67001f6283.slice/crio-a89078e1b6b81b5fbbad66859b3c17625cda152ab308f6d94e02cba957a2efee WatchSource:0}: Error finding container a89078e1b6b81b5fbbad66859b3c17625cda152ab308f6d94e02cba957a2efee: Status 404 returned error can't find the container with id a89078e1b6b81b5fbbad66859b3c17625cda152ab308f6d94e02cba957a2efee Jul 29 14:30:10.218383 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:10.218344 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" event={"ID":"fb22f637-9157-4c15-8706-2c67001f6283","Type":"ContainerStarted","Data":"805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78"} Jul 29 14:30:10.218893 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:10.218410 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" event={"ID":"fb22f637-9157-4c15-8706-2c67001f6283","Type":"ContainerStarted","Data":"a89078e1b6b81b5fbbad66859b3c17625cda152ab308f6d94e02cba957a2efee"} Jul 29 14:30:10.218893 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:10.218453 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:10.220104 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:10.220070 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" event={"ID":"69222735-bb93-4e8d-ad2f-a67133cc8325","Type":"ContainerStarted","Data":"858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c"} Jul 29 14:30:10.220221 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:10.220129 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" event={"ID":"69222735-bb93-4e8d-ad2f-a67133cc8325","Type":"ContainerStarted","Data":"c727d32e1cb4bcd2167cdfeaa3e8de905dc33d2a1fe232b1606bec37abaa34b3"} Jul 29 14:30:10.220221 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:10.220150 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:10.240885 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:10.240812 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" podStartSLOduration=2.240787614 podStartE2EDuration="2.240787614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:30:10.237721248 +0000 UTC m=+1735.890315107" watchObservedRunningTime="2026-07-29 14:30:10.240787614 +0000 UTC m=+1735.893381450" Jul 29 14:30:10.257840 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:10.257773 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" podStartSLOduration=2.257754179 podStartE2EDuration="2.257754179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:30:10.255730413 +0000 UTC m=+1735.908324248" watchObservedRunningTime="2026-07-29 14:30:10.257754179 +0000 UTC m=+1735.910348050" Jul 29 14:30:21.227224 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:21.227191 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:21.227744 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:21.227269 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:38.626763 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.626722 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj"] Jul 29 14:30:38.627436 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.626951 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" podUID="69222735-bb93-4e8d-ad2f-a67133cc8325" containerName="payload-processing" containerID="cri-o://858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c" gracePeriod=30 Jul 29 14:30:38.631279 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.631243 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt"] Jul 29 14:30:38.631815 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.631753 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" podUID="fb22f637-9157-4c15-8706-2c67001f6283" containerName="payload-pre-processing" containerID="cri-o://805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78" gracePeriod=30 Jul 29 14:30:38.930630 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.930596 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:38.933751 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.933725 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:38.977251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.977217 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljlkd\" (UniqueName: \"kubernetes.io/projected/fb22f637-9157-4c15-8706-2c67001f6283-kube-api-access-ljlkd\") pod \"fb22f637-9157-4c15-8706-2c67001f6283\" (UID: \"fb22f637-9157-4c15-8706-2c67001f6283\") " Jul 29 14:30:38.977251 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.977264 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc4vn\" (UniqueName: \"kubernetes.io/projected/69222735-bb93-4e8d-ad2f-a67133cc8325-kube-api-access-vc4vn\") pod \"69222735-bb93-4e8d-ad2f-a67133cc8325\" (UID: \"69222735-bb93-4e8d-ad2f-a67133cc8325\") " Jul 29 14:30:38.977522 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.977368 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb22f637-9157-4c15-8706-2c67001f6283-plugins-config-volume\") pod \"fb22f637-9157-4c15-8706-2c67001f6283\" (UID: \"fb22f637-9157-4c15-8706-2c67001f6283\") " Jul 29 14:30:38.977522 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.977456 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69222735-bb93-4e8d-ad2f-a67133cc8325-plugins-config-volume\") pod \"69222735-bb93-4e8d-ad2f-a67133cc8325\" (UID: \"69222735-bb93-4e8d-ad2f-a67133cc8325\") " Jul 29 14:30:38.977863 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.977833 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69222735-bb93-4e8d-ad2f-a67133cc8325-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69222735-bb93-4e8d-ad2f-a67133cc8325" (UID: "69222735-bb93-4e8d-ad2f-a67133cc8325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:30:38.978002 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.977871 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb22f637-9157-4c15-8706-2c67001f6283-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb22f637-9157-4c15-8706-2c67001f6283" (UID: "fb22f637-9157-4c15-8706-2c67001f6283"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:30:38.979723 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.979695 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69222735-bb93-4e8d-ad2f-a67133cc8325-kube-api-access-vc4vn" (OuterVolumeSpecName: "kube-api-access-vc4vn") pod "69222735-bb93-4e8d-ad2f-a67133cc8325" (UID: "69222735-bb93-4e8d-ad2f-a67133cc8325"). InnerVolumeSpecName "kube-api-access-vc4vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:30:38.979825 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:38.979732 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb22f637-9157-4c15-8706-2c67001f6283-kube-api-access-ljlkd" (OuterVolumeSpecName: "kube-api-access-ljlkd") pod "fb22f637-9157-4c15-8706-2c67001f6283" (UID: "fb22f637-9157-4c15-8706-2c67001f6283"). InnerVolumeSpecName "kube-api-access-ljlkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:30:39.078761 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.078718 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb22f637-9157-4c15-8706-2c67001f6283-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:30:39.078761 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.078751 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69222735-bb93-4e8d-ad2f-a67133cc8325-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:30:39.078761 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.078762 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljlkd\" (UniqueName: \"kubernetes.io/projected/fb22f637-9157-4c15-8706-2c67001f6283-kube-api-access-ljlkd\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:30:39.078761 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.078772 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc4vn\" (UniqueName: \"kubernetes.io/projected/69222735-bb93-4e8d-ad2f-a67133cc8325-kube-api-access-vc4vn\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:30:39.365243 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.365202 2549 generic.go:356] "Generic (PLEG): container finished" podID="fb22f637-9157-4c15-8706-2c67001f6283" containerID="805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78" exitCode=0 Jul 29 14:30:39.365445 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.365275 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" Jul 29 14:30:39.365445 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.365291 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" event={"ID":"fb22f637-9157-4c15-8706-2c67001f6283","Type":"ContainerDied","Data":"805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78"} Jul 29 14:30:39.365445 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.365412 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt" event={"ID":"fb22f637-9157-4c15-8706-2c67001f6283","Type":"ContainerDied","Data":"a89078e1b6b81b5fbbad66859b3c17625cda152ab308f6d94e02cba957a2efee"} Jul 29 14:30:39.365445 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.365442 2549 scope.go:117] "RemoveContainer" containerID="805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78" Jul 29 14:30:39.366886 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.366861 2549 generic.go:356] "Generic (PLEG): container finished" podID="69222735-bb93-4e8d-ad2f-a67133cc8325" containerID="858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c" exitCode=0 Jul 29 14:30:39.366979 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.366917 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" event={"ID":"69222735-bb93-4e8d-ad2f-a67133cc8325","Type":"ContainerDied","Data":"858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c"} Jul 29 14:30:39.366979 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.366924 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" Jul 29 14:30:39.366979 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.366966 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj" event={"ID":"69222735-bb93-4e8d-ad2f-a67133cc8325","Type":"ContainerDied","Data":"c727d32e1cb4bcd2167cdfeaa3e8de905dc33d2a1fe232b1606bec37abaa34b3"} Jul 29 14:30:39.376185 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.376160 2549 scope.go:117] "RemoveContainer" containerID="805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78" Jul 29 14:30:39.376623 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:30:39.376592 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78\": container with ID starting with 805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78 not found: ID does not exist" containerID="805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78" Jul 29 14:30:39.376727 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.376638 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78"} err="failed to get container status \"805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78\": rpc error: code = NotFound desc = could not find container \"805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78\": container with ID starting with 805c955620c9d6b6df5494e3cc2ae3c1799f0d737e81fac512e8fe2276b25e78 not found: ID does not exist" Jul 29 14:30:39.376727 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.376666 2549 scope.go:117] "RemoveContainer" containerID="858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c" Jul 29 14:30:39.385582 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.385554 2549 scope.go:117] "RemoveContainer" containerID="858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c" Jul 29 14:30:39.385915 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:30:39.385894 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c\": container with ID starting with 858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c not found: ID does not exist" containerID="858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c" Jul 29 14:30:39.385977 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.385922 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c"} err="failed to get container status \"858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c\": rpc error: code = NotFound desc = could not find container \"858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c\": container with ID starting with 858bb5bccc6e2b6172c6842e83d0231036f5c7a775ba4765f7c72d3c10ac4e5c not found: ID does not exist" Jul 29 14:30:39.389802 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.389770 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt"] Jul 29 14:30:39.394044 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.394017 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee08ac7d-66bf8f57b8-rfzpt"] Jul 29 14:30:39.404755 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.404718 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj"] Jul 29 14:30:39.408690 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:39.408658 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee08ac7d-747bbff9d-nz2zj"] Jul 29 14:30:40.865906 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:40.865861 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69222735-bb93-4e8d-ad2f-a67133cc8325" path="/var/lib/kubelet/pods/69222735-bb93-4e8d-ad2f-a67133cc8325/volumes" Jul 29 14:30:40.866366 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:40.866356 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb22f637-9157-4c15-8706-2c67001f6283" path="/var/lib/kubelet/pods/fb22f637-9157-4c15-8706-2c67001f6283/volumes" Jul 29 14:30:44.373060 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.373021 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l"] Jul 29 14:30:44.373589 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.373421 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb22f637-9157-4c15-8706-2c67001f6283" containerName="payload-pre-processing" Jul 29 14:30:44.373589 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.373434 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb22f637-9157-4c15-8706-2c67001f6283" containerName="payload-pre-processing" Jul 29 14:30:44.373589 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.373450 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69222735-bb93-4e8d-ad2f-a67133cc8325" containerName="payload-processing" Jul 29 14:30:44.373589 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.373456 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="69222735-bb93-4e8d-ad2f-a67133cc8325" containerName="payload-processing" Jul 29 14:30:44.373589 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.373512 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="69222735-bb93-4e8d-ad2f-a67133cc8325" containerName="payload-processing" Jul 29 14:30:44.373589 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.373524 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb22f637-9157-4c15-8706-2c67001f6283" containerName="payload-pre-processing" Jul 29 14:30:44.378094 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.378070 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:30:44.381070 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.381038 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-330348dc\"" Jul 29 14:30:44.381260 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.381120 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-330348dc-dockercfg-25289\"" Jul 29 14:30:44.391960 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.391926 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l"] Jul 29 14:30:44.402469 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.402432 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz"] Jul 29 14:30:44.411064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.408071 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:44.427730 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.427513 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz"] Jul 29 14:30:44.429130 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.429096 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l\" (UID: \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:30:44.429280 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.429159 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j95nd\" (UniqueName: \"kubernetes.io/projected/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-kube-api-access-j95nd\") pod \"payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l\" (UID: \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:30:44.530590 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.530537 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2vdh\" (UniqueName: \"kubernetes.io/projected/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-kube-api-access-g2vdh\") pod \"payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz\" (UID: \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\") " pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:44.530590 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.530599 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l\" (UID: \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:30:44.530883 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.530651 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j95nd\" (UniqueName: \"kubernetes.io/projected/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-kube-api-access-j95nd\") pod \"payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l\" (UID: \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:30:44.530883 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.530698 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-plugins-config-volume\") pod \"payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz\" (UID: \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\") " pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:44.531444 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.531420 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l\" (UID: \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:30:44.538974 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.538943 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j95nd\" (UniqueName: \"kubernetes.io/projected/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-kube-api-access-j95nd\") pod \"payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l\" (UID: \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:30:44.631307 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.631200 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2vdh\" (UniqueName: \"kubernetes.io/projected/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-kube-api-access-g2vdh\") pod \"payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz\" (UID: \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\") " pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:44.631307 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.631296 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-plugins-config-volume\") pod \"payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz\" (UID: \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\") " pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:44.632134 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.632105 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-plugins-config-volume\") pod \"payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz\" (UID: \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\") " pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:44.647069 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.647038 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2vdh\" (UniqueName: \"kubernetes.io/projected/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-kube-api-access-g2vdh\") pod \"payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz\" (UID: \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\") " pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:44.690032 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.689989 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:30:44.728777 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.728736 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:44.853092 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.852998 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l"] Jul 29 14:30:44.857377 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:30:44.857306 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1301e32_4d36_4a1f_bc21_e4f77b7f765d.slice/crio-12fbbaac0a379d18bcd6bdf91811903e6750d8e36e11ced6cc389142f38711f7 WatchSource:0}: Error finding container 12fbbaac0a379d18bcd6bdf91811903e6750d8e36e11ced6cc389142f38711f7: Status 404 returned error can't find the container with id 12fbbaac0a379d18bcd6bdf91811903e6750d8e36e11ced6cc389142f38711f7 Jul 29 14:30:44.898673 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:44.898645 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz"] Jul 29 14:30:44.900857 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:30:44.900827 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73907b9_50b5_4fb8_a65e_8a9de9493e4a.slice/crio-e0d8ae7111c5e9902e26a7677716f8afd752df70928432d2891b7a3a5ac25ae7 WatchSource:0}: Error finding container e0d8ae7111c5e9902e26a7677716f8afd752df70928432d2891b7a3a5ac25ae7: Status 404 returned error can't find the container with id e0d8ae7111c5e9902e26a7677716f8afd752df70928432d2891b7a3a5ac25ae7 Jul 29 14:30:45.406254 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:45.406211 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" event={"ID":"b73907b9-50b5-4fb8-a65e-8a9de9493e4a","Type":"ContainerStarted","Data":"68213f8bc37abdbe50d50b5f8ba795f44885fa6779255c76eea81e47570a54f6"} Jul 29 14:30:45.406779 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:45.406275 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" event={"ID":"b73907b9-50b5-4fb8-a65e-8a9de9493e4a","Type":"ContainerStarted","Data":"e0d8ae7111c5e9902e26a7677716f8afd752df70928432d2891b7a3a5ac25ae7"} Jul 29 14:30:45.406779 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:45.406326 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:45.408093 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:45.408055 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" event={"ID":"d1301e32-4d36-4a1f-bc21-e4f77b7f765d","Type":"ContainerStarted","Data":"d3dd752f6bbf17ec8cb71302e96ef2164417d22de6b87ed76c4e68595d9528c9"} Jul 29 14:30:45.408209 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:45.408124 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:30:45.408209 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:45.408143 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" event={"ID":"d1301e32-4d36-4a1f-bc21-e4f77b7f765d","Type":"ContainerStarted","Data":"12fbbaac0a379d18bcd6bdf91811903e6750d8e36e11ced6cc389142f38711f7"} Jul 29 14:30:45.422659 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:45.422589 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" podStartSLOduration=1.422545242 podStartE2EDuration="1.422545242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:30:45.421773113 +0000 UTC m=+1771.074366953" watchObservedRunningTime="2026-07-29 14:30:45.422545242 +0000 UTC m=+1771.075139076" Jul 29 14:30:45.438406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:45.438340 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" podStartSLOduration=1.438296595 podStartE2EDuration="1.438296595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:30:45.436363309 +0000 UTC m=+1771.088957166" watchObservedRunningTime="2026-07-29 14:30:45.438296595 +0000 UTC m=+1771.090890433" Jul 29 14:30:56.415694 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:56.415658 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:30:56.416295 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:30:56.415746 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:31:14.246474 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.246428 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz"] Jul 29 14:31:14.247013 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.246774 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" podUID="b73907b9-50b5-4fb8-a65e-8a9de9493e4a" containerName="payload-processing" containerID="cri-o://68213f8bc37abdbe50d50b5f8ba795f44885fa6779255c76eea81e47570a54f6" gracePeriod=30 Jul 29 14:31:14.254754 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.254710 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l"] Jul 29 14:31:14.255213 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.255014 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" podUID="d1301e32-4d36-4a1f-bc21-e4f77b7f765d" containerName="payload-pre-processing" containerID="cri-o://d3dd752f6bbf17ec8cb71302e96ef2164417d22de6b87ed76c4e68595d9528c9" gracePeriod=30 Jul 29 14:31:14.557427 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.557393 2549 generic.go:356] "Generic (PLEG): container finished" podID="b73907b9-50b5-4fb8-a65e-8a9de9493e4a" containerID="68213f8bc37abdbe50d50b5f8ba795f44885fa6779255c76eea81e47570a54f6" exitCode=0 Jul 29 14:31:14.557615 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.557470 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" event={"ID":"b73907b9-50b5-4fb8-a65e-8a9de9493e4a","Type":"ContainerDied","Data":"68213f8bc37abdbe50d50b5f8ba795f44885fa6779255c76eea81e47570a54f6"} Jul 29 14:31:14.557615 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.557533 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" event={"ID":"b73907b9-50b5-4fb8-a65e-8a9de9493e4a","Type":"ContainerDied","Data":"e0d8ae7111c5e9902e26a7677716f8afd752df70928432d2891b7a3a5ac25ae7"} Jul 29 14:31:14.557615 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.557554 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0d8ae7111c5e9902e26a7677716f8afd752df70928432d2891b7a3a5ac25ae7" Jul 29 14:31:14.559122 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.559093 2549 generic.go:356] "Generic (PLEG): container finished" podID="d1301e32-4d36-4a1f-bc21-e4f77b7f765d" containerID="d3dd752f6bbf17ec8cb71302e96ef2164417d22de6b87ed76c4e68595d9528c9" exitCode=0 Jul 29 14:31:14.559270 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.559152 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" event={"ID":"d1301e32-4d36-4a1f-bc21-e4f77b7f765d","Type":"ContainerDied","Data":"d3dd752f6bbf17ec8cb71302e96ef2164417d22de6b87ed76c4e68595d9528c9"} Jul 29 14:31:14.559270 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.559209 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" event={"ID":"d1301e32-4d36-4a1f-bc21-e4f77b7f765d","Type":"ContainerDied","Data":"12fbbaac0a379d18bcd6bdf91811903e6750d8e36e11ced6cc389142f38711f7"} Jul 29 14:31:14.559270 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.559231 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12fbbaac0a379d18bcd6bdf91811903e6750d8e36e11ced6cc389142f38711f7" Jul 29 14:31:14.569739 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.569707 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:31:14.573209 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.573178 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:31:14.628595 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.628561 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j95nd\" (UniqueName: \"kubernetes.io/projected/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-kube-api-access-j95nd\") pod \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\" (UID: \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\") " Jul 29 14:31:14.628803 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.628655 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-plugins-config-volume\") pod \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\" (UID: \"d1301e32-4d36-4a1f-bc21-e4f77b7f765d\") " Jul 29 14:31:14.628803 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.628715 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2vdh\" (UniqueName: \"kubernetes.io/projected/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-kube-api-access-g2vdh\") pod \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\" (UID: \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\") " Jul 29 14:31:14.628803 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.628736 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-plugins-config-volume\") pod \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\" (UID: \"b73907b9-50b5-4fb8-a65e-8a9de9493e4a\") " Jul 29 14:31:14.629245 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.629201 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1301e32-4d36-4a1f-bc21-e4f77b7f765d" (UID: "d1301e32-4d36-4a1f-bc21-e4f77b7f765d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:31:14.629398 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.629211 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b73907b9-50b5-4fb8-a65e-8a9de9493e4a" (UID: "b73907b9-50b5-4fb8-a65e-8a9de9493e4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:31:14.631043 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.631015 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-kube-api-access-g2vdh" (OuterVolumeSpecName: "kube-api-access-g2vdh") pod "b73907b9-50b5-4fb8-a65e-8a9de9493e4a" (UID: "b73907b9-50b5-4fb8-a65e-8a9de9493e4a"). InnerVolumeSpecName "kube-api-access-g2vdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:31:14.631123 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.631057 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-kube-api-access-j95nd" (OuterVolumeSpecName: "kube-api-access-j95nd") pod "d1301e32-4d36-4a1f-bc21-e4f77b7f765d" (UID: "d1301e32-4d36-4a1f-bc21-e4f77b7f765d"). InnerVolumeSpecName "kube-api-access-j95nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:31:14.729544 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.729498 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:31:14.729544 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.729530 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2vdh\" (UniqueName: \"kubernetes.io/projected/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-kube-api-access-g2vdh\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:31:14.729544 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.729544 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73907b9-50b5-4fb8-a65e-8a9de9493e4a-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:31:14.729544 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.729555 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j95nd\" (UniqueName: \"kubernetes.io/projected/d1301e32-4d36-4a1f-bc21-e4f77b7f765d-kube-api-access-j95nd\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:31:14.899014 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.898976 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:31:14.900805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:14.900778 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:31:15.565381 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:15.565249 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l" Jul 29 14:31:15.565381 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:15.565275 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz" Jul 29 14:31:15.583507 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:15.583464 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l"] Jul 29 14:31:15.587684 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:15.587648 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-330348dc-7598ddfcfb-mw45l"] Jul 29 14:31:15.604947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:15.604906 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz"] Jul 29 14:31:15.606903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:15.606871 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-330348dc-5544d9fcfd-wntkz"] Jul 29 14:31:16.864846 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:16.864809 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b73907b9-50b5-4fb8-a65e-8a9de9493e4a" path="/var/lib/kubelet/pods/b73907b9-50b5-4fb8-a65e-8a9de9493e4a/volumes" Jul 29 14:31:16.865278 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:31:16.865195 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1301e32-4d36-4a1f-bc21-e4f77b7f765d" path="/var/lib/kubelet/pods/d1301e32-4d36-4a1f-bc21-e4f77b7f765d/volumes" Jul 29 14:33:01.775384 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.775281 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf"] Jul 29 14:33:01.775903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.775709 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1301e32-4d36-4a1f-bc21-e4f77b7f765d" containerName="payload-pre-processing" Jul 29 14:33:01.775903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.775723 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1301e32-4d36-4a1f-bc21-e4f77b7f765d" containerName="payload-pre-processing" Jul 29 14:33:01.775903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.775745 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b73907b9-50b5-4fb8-a65e-8a9de9493e4a" containerName="payload-processing" Jul 29 14:33:01.775903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.775752 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b73907b9-50b5-4fb8-a65e-8a9de9493e4a" containerName="payload-processing" Jul 29 14:33:01.775903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.775810 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1301e32-4d36-4a1f-bc21-e4f77b7f765d" containerName="payload-pre-processing" Jul 29 14:33:01.775903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.775821 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b73907b9-50b5-4fb8-a65e-8a9de9493e4a" containerName="payload-processing" Jul 29 14:33:01.778408 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.778389 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:01.780976 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.780949 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-10f19eac\"" Jul 29 14:33:01.788882 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.788847 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf"] Jul 29 14:33:01.843202 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.843160 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh"] Jul 29 14:33:01.846734 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.846712 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:01.847179 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.847156 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f3879-6d7a-43c6-9c65-6127652b4649-plugins-config-volume\") pod \"payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf\" (UID: \"ac8f3879-6d7a-43c6-9c65-6127652b4649\") " pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:01.847287 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.847271 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qftwf\" (UniqueName: \"kubernetes.io/projected/ac8f3879-6d7a-43c6-9c65-6127652b4649-kube-api-access-qftwf\") pod \"payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf\" (UID: \"ac8f3879-6d7a-43c6-9c65-6127652b4649\") " pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:01.850457 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.850426 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-10f19eac-dockercfg-8rdt9\"" Jul 29 14:33:01.862445 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.862413 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh"] Jul 29 14:33:01.948927 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.948878 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg6qk\" (UniqueName: \"kubernetes.io/projected/a027e178-55d4-4163-9f46-5a608a1bf771-kube-api-access-zg6qk\") pod \"payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh\" (UID: \"a027e178-55d4-4163-9f46-5a608a1bf771\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:01.949163 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.948941 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f3879-6d7a-43c6-9c65-6127652b4649-plugins-config-volume\") pod \"payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf\" (UID: \"ac8f3879-6d7a-43c6-9c65-6127652b4649\") " pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:01.949244 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.949179 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qftwf\" (UniqueName: \"kubernetes.io/projected/ac8f3879-6d7a-43c6-9c65-6127652b4649-kube-api-access-qftwf\") pod \"payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf\" (UID: \"ac8f3879-6d7a-43c6-9c65-6127652b4649\") " pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:01.949244 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.949216 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a027e178-55d4-4163-9f46-5a608a1bf771-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh\" (UID: \"a027e178-55d4-4163-9f46-5a608a1bf771\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:01.949745 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.949720 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f3879-6d7a-43c6-9c65-6127652b4649-plugins-config-volume\") pod \"payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf\" (UID: \"ac8f3879-6d7a-43c6-9c65-6127652b4649\") " pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:01.957872 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:01.957830 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qftwf\" (UniqueName: \"kubernetes.io/projected/ac8f3879-6d7a-43c6-9c65-6127652b4649-kube-api-access-qftwf\") pod \"payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf\" (UID: \"ac8f3879-6d7a-43c6-9c65-6127652b4649\") " pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:02.050726 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:02.050619 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a027e178-55d4-4163-9f46-5a608a1bf771-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh\" (UID: \"a027e178-55d4-4163-9f46-5a608a1bf771\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:02.050726 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:02.050701 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg6qk\" (UniqueName: \"kubernetes.io/projected/a027e178-55d4-4163-9f46-5a608a1bf771-kube-api-access-zg6qk\") pod \"payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh\" (UID: \"a027e178-55d4-4163-9f46-5a608a1bf771\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:02.051467 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:02.051447 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a027e178-55d4-4163-9f46-5a608a1bf771-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh\" (UID: \"a027e178-55d4-4163-9f46-5a608a1bf771\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:02.061056 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:02.061025 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg6qk\" (UniqueName: \"kubernetes.io/projected/a027e178-55d4-4163-9f46-5a608a1bf771-kube-api-access-zg6qk\") pod \"payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh\" (UID: \"a027e178-55d4-4163-9f46-5a608a1bf771\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:02.089873 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:02.089820 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:02.157649 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:02.157598 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:02.255897 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:02.255739 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf"] Jul 29 14:33:02.272632 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:33:02.269369 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8f3879_6d7a_43c6_9c65_6127652b4649.slice/crio-5e19a4ad27580a710b00bfe0b05e3924e89321e807f3bda190665ab35615558f WatchSource:0}: Error finding container 5e19a4ad27580a710b00bfe0b05e3924e89321e807f3bda190665ab35615558f: Status 404 returned error can't find the container with id 5e19a4ad27580a710b00bfe0b05e3924e89321e807f3bda190665ab35615558f Jul 29 14:33:02.328405 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:02.328375 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh"] Jul 29 14:33:02.330471 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:33:02.330424 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda027e178_55d4_4163_9f46_5a608a1bf771.slice/crio-c19d2f624c91166e8583cbab35a6f5dca147623fd787fdbccc76edeb07c5dfa4 WatchSource:0}: Error finding container c19d2f624c91166e8583cbab35a6f5dca147623fd787fdbccc76edeb07c5dfa4: Status 404 returned error can't find the container with id c19d2f624c91166e8583cbab35a6f5dca147623fd787fdbccc76edeb07c5dfa4 Jul 29 14:33:03.097389 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:03.097338 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" event={"ID":"a027e178-55d4-4163-9f46-5a608a1bf771","Type":"ContainerStarted","Data":"6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a"} Jul 29 14:33:03.098213 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:03.097410 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" event={"ID":"a027e178-55d4-4163-9f46-5a608a1bf771","Type":"ContainerStarted","Data":"c19d2f624c91166e8583cbab35a6f5dca147623fd787fdbccc76edeb07c5dfa4"} Jul 29 14:33:03.098213 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:03.097445 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:03.099067 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:03.099031 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" event={"ID":"ac8f3879-6d7a-43c6-9c65-6127652b4649","Type":"ContainerStarted","Data":"fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599"} Jul 29 14:33:03.099188 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:03.099086 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" event={"ID":"ac8f3879-6d7a-43c6-9c65-6127652b4649","Type":"ContainerStarted","Data":"5e19a4ad27580a710b00bfe0b05e3924e89321e807f3bda190665ab35615558f"} Jul 29 14:33:03.099188 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:03.099116 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:03.114830 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:03.114772 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" podStartSLOduration=2.114755476 podStartE2EDuration="2.114755476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:33:03.11377123 +0000 UTC m=+1908.766365069" watchObservedRunningTime="2026-07-29 14:33:03.114755476 +0000 UTC m=+1908.767349311" Jul 29 14:33:03.131973 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:03.131907 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" podStartSLOduration=2.131884729 podStartE2EDuration="2.131884729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:33:03.130830421 +0000 UTC m=+1908.783424257" watchObservedRunningTime="2026-07-29 14:33:03.131884729 +0000 UTC m=+1908.784478563" Jul 29 14:33:14.106414 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:14.106381 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:14.106875 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:14.106489 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:43.435628 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.435547 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf"] Jul 29 14:33:43.436348 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.435837 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" podUID="ac8f3879-6d7a-43c6-9c65-6127652b4649" containerName="payload-processing" containerID="cri-o://fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599" gracePeriod=30 Jul 29 14:33:43.444533 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.444289 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh"] Jul 29 14:33:43.444870 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.444756 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" podUID="a027e178-55d4-4163-9f46-5a608a1bf771" containerName="payload-pre-processing" containerID="cri-o://6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a" gracePeriod=30 Jul 29 14:33:43.743950 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.743920 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:43.747049 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.747027 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:43.861997 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.861959 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f3879-6d7a-43c6-9c65-6127652b4649-plugins-config-volume\") pod \"ac8f3879-6d7a-43c6-9c65-6127652b4649\" (UID: \"ac8f3879-6d7a-43c6-9c65-6127652b4649\") " Jul 29 14:33:43.862235 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.862034 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg6qk\" (UniqueName: \"kubernetes.io/projected/a027e178-55d4-4163-9f46-5a608a1bf771-kube-api-access-zg6qk\") pod \"a027e178-55d4-4163-9f46-5a608a1bf771\" (UID: \"a027e178-55d4-4163-9f46-5a608a1bf771\") " Jul 29 14:33:43.862235 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.862068 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qftwf\" (UniqueName: \"kubernetes.io/projected/ac8f3879-6d7a-43c6-9c65-6127652b4649-kube-api-access-qftwf\") pod \"ac8f3879-6d7a-43c6-9c65-6127652b4649\" (UID: \"ac8f3879-6d7a-43c6-9c65-6127652b4649\") " Jul 29 14:33:43.862235 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.862106 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a027e178-55d4-4163-9f46-5a608a1bf771-plugins-config-volume\") pod \"a027e178-55d4-4163-9f46-5a608a1bf771\" (UID: \"a027e178-55d4-4163-9f46-5a608a1bf771\") " Jul 29 14:33:43.862526 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.862492 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8f3879-6d7a-43c6-9c65-6127652b4649-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac8f3879-6d7a-43c6-9c65-6127652b4649" (UID: "ac8f3879-6d7a-43c6-9c65-6127652b4649"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:33:43.862604 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.862566 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a027e178-55d4-4163-9f46-5a608a1bf771-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a027e178-55d4-4163-9f46-5a608a1bf771" (UID: "a027e178-55d4-4163-9f46-5a608a1bf771"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:33:43.864911 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.864878 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a027e178-55d4-4163-9f46-5a608a1bf771-kube-api-access-zg6qk" (OuterVolumeSpecName: "kube-api-access-zg6qk") pod "a027e178-55d4-4163-9f46-5a608a1bf771" (UID: "a027e178-55d4-4163-9f46-5a608a1bf771"). InnerVolumeSpecName "kube-api-access-zg6qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:33:43.864911 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.864887 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f3879-6d7a-43c6-9c65-6127652b4649-kube-api-access-qftwf" (OuterVolumeSpecName: "kube-api-access-qftwf") pod "ac8f3879-6d7a-43c6-9c65-6127652b4649" (UID: "ac8f3879-6d7a-43c6-9c65-6127652b4649"). InnerVolumeSpecName "kube-api-access-qftwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:33:43.963798 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.963697 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg6qk\" (UniqueName: \"kubernetes.io/projected/a027e178-55d4-4163-9f46-5a608a1bf771-kube-api-access-zg6qk\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:33:43.963798 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.963731 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qftwf\" (UniqueName: \"kubernetes.io/projected/ac8f3879-6d7a-43c6-9c65-6127652b4649-kube-api-access-qftwf\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:33:43.963798 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.963741 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a027e178-55d4-4163-9f46-5a608a1bf771-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:33:43.963798 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:43.963749 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f3879-6d7a-43c6-9c65-6127652b4649-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:33:44.311254 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.311215 2549 generic.go:356] "Generic (PLEG): container finished" podID="a027e178-55d4-4163-9f46-5a608a1bf771" containerID="6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a" exitCode=0 Jul 29 14:33:44.311471 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.311265 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" Jul 29 14:33:44.311471 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.311296 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" event={"ID":"a027e178-55d4-4163-9f46-5a608a1bf771","Type":"ContainerDied","Data":"6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a"} Jul 29 14:33:44.311471 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.311380 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh" event={"ID":"a027e178-55d4-4163-9f46-5a608a1bf771","Type":"ContainerDied","Data":"c19d2f624c91166e8583cbab35a6f5dca147623fd787fdbccc76edeb07c5dfa4"} Jul 29 14:33:44.311471 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.311411 2549 scope.go:117] "RemoveContainer" containerID="6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a" Jul 29 14:33:44.312746 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.312720 2549 generic.go:356] "Generic (PLEG): container finished" podID="ac8f3879-6d7a-43c6-9c65-6127652b4649" containerID="fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599" exitCode=0 Jul 29 14:33:44.312871 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.312756 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" event={"ID":"ac8f3879-6d7a-43c6-9c65-6127652b4649","Type":"ContainerDied","Data":"fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599"} Jul 29 14:33:44.312871 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.312775 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" Jul 29 14:33:44.312871 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.312796 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf" event={"ID":"ac8f3879-6d7a-43c6-9c65-6127652b4649","Type":"ContainerDied","Data":"5e19a4ad27580a710b00bfe0b05e3924e89321e807f3bda190665ab35615558f"} Jul 29 14:33:44.321344 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.321306 2549 scope.go:117] "RemoveContainer" containerID="6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a" Jul 29 14:33:44.321686 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:33:44.321662 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a\": container with ID starting with 6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a not found: ID does not exist" containerID="6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a" Jul 29 14:33:44.321827 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.321693 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a"} err="failed to get container status \"6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a\": rpc error: code = NotFound desc = could not find container \"6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a\": container with ID starting with 6f2654fc26f23c6d55b651da508c98795a76bc93b8703c346f3b93c89478086a not found: ID does not exist" Jul 29 14:33:44.321827 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.321712 2549 scope.go:117] "RemoveContainer" containerID="fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599" Jul 29 14:33:44.333760 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.333732 2549 scope.go:117] "RemoveContainer" containerID="fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599" Jul 29 14:33:44.334084 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:33:44.334063 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599\": container with ID starting with fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599 not found: ID does not exist" containerID="fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599" Jul 29 14:33:44.334158 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.334091 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599"} err="failed to get container status \"fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599\": rpc error: code = NotFound desc = could not find container \"fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599\": container with ID starting with fdf9ce7770d2b207fe7eaa75a35ad42dc3504cbe7342908cb3d12c6ae81b9599 not found: ID does not exist" Jul 29 14:33:44.338459 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.338424 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf"] Jul 29 14:33:44.341068 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.341038 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-10f19eac-84dfb95ffd-f2gwf"] Jul 29 14:33:44.350736 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.350703 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh"] Jul 29 14:33:44.354428 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.354396 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-10f19eac-6c5ccbdbb8-jddgh"] Jul 29 14:33:44.866639 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.866604 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a027e178-55d4-4163-9f46-5a608a1bf771" path="/var/lib/kubelet/pods/a027e178-55d4-4163-9f46-5a608a1bf771/volumes" Jul 29 14:33:44.867061 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:33:44.866981 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac8f3879-6d7a-43c6-9c65-6127652b4649" path="/var/lib/kubelet/pods/ac8f3879-6d7a-43c6-9c65-6127652b4649/volumes" Jul 29 14:34:13.749913 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.749864 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2"] Jul 29 14:34:13.750430 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.750253 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac8f3879-6d7a-43c6-9c65-6127652b4649" containerName="payload-processing" Jul 29 14:34:13.750430 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.750265 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac8f3879-6d7a-43c6-9c65-6127652b4649" containerName="payload-processing" Jul 29 14:34:13.750430 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.750272 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a027e178-55d4-4163-9f46-5a608a1bf771" containerName="payload-pre-processing" Jul 29 14:34:13.750430 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.750278 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a027e178-55d4-4163-9f46-5a608a1bf771" containerName="payload-pre-processing" Jul 29 14:34:13.750430 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.750361 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a027e178-55d4-4163-9f46-5a608a1bf771" containerName="payload-pre-processing" Jul 29 14:34:13.750430 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.750370 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac8f3879-6d7a-43c6-9c65-6127652b4649" containerName="payload-processing" Jul 29 14:34:13.753592 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.753572 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:34:13.756530 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.756506 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c3fb6290\"" Jul 29 14:34:13.763336 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.763262 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2"] Jul 29 14:34:13.774423 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.774381 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m"] Jul 29 14:34:13.777949 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.777922 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:13.780470 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.780445 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c3fb6290-dockercfg-96wrl\"" Jul 29 14:34:13.789414 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.789384 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m"] Jul 29 14:34:13.851604 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.851522 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxbb\" (UniqueName: \"kubernetes.io/projected/2f856065-f8f0-428e-92a6-bdd6ec801db8-kube-api-access-qjxbb\") pod \"payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2\" (UID: \"2f856065-f8f0-428e-92a6-bdd6ec801db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:34:13.851802 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.851634 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243db38-e112-4573-b8b2-beb11309dc71-plugins-config-volume\") pod \"payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m\" (UID: \"2243db38-e112-4573-b8b2-beb11309dc71\") " pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:13.851802 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.851667 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f856065-f8f0-428e-92a6-bdd6ec801db8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2\" (UID: \"2f856065-f8f0-428e-92a6-bdd6ec801db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:34:13.851802 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.851760 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgs49\" (UniqueName: \"kubernetes.io/projected/2243db38-e112-4573-b8b2-beb11309dc71-kube-api-access-kgs49\") pod \"payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m\" (UID: \"2243db38-e112-4573-b8b2-beb11309dc71\") " pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:13.952944 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.952900 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f856065-f8f0-428e-92a6-bdd6ec801db8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2\" (UID: \"2f856065-f8f0-428e-92a6-bdd6ec801db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:34:13.953190 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.952990 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgs49\" (UniqueName: \"kubernetes.io/projected/2243db38-e112-4573-b8b2-beb11309dc71-kube-api-access-kgs49\") pod \"payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m\" (UID: \"2243db38-e112-4573-b8b2-beb11309dc71\") " pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:13.953190 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.953030 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxbb\" (UniqueName: \"kubernetes.io/projected/2f856065-f8f0-428e-92a6-bdd6ec801db8-kube-api-access-qjxbb\") pod \"payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2\" (UID: \"2f856065-f8f0-428e-92a6-bdd6ec801db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:34:13.953190 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.953069 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243db38-e112-4573-b8b2-beb11309dc71-plugins-config-volume\") pod \"payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m\" (UID: \"2243db38-e112-4573-b8b2-beb11309dc71\") " pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:13.953842 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.953813 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f856065-f8f0-428e-92a6-bdd6ec801db8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2\" (UID: \"2f856065-f8f0-428e-92a6-bdd6ec801db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:34:13.953842 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.953833 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243db38-e112-4573-b8b2-beb11309dc71-plugins-config-volume\") pod \"payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m\" (UID: \"2243db38-e112-4573-b8b2-beb11309dc71\") " pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:13.962579 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.962541 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxbb\" (UniqueName: \"kubernetes.io/projected/2f856065-f8f0-428e-92a6-bdd6ec801db8-kube-api-access-qjxbb\") pod \"payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2\" (UID: \"2f856065-f8f0-428e-92a6-bdd6ec801db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:34:13.962579 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:13.962564 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgs49\" (UniqueName: \"kubernetes.io/projected/2243db38-e112-4573-b8b2-beb11309dc71-kube-api-access-kgs49\") pod \"payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m\" (UID: \"2243db38-e112-4573-b8b2-beb11309dc71\") " pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:14.066662 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.066562 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:34:14.091503 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.091461 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:14.230401 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.230367 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2"] Jul 29 14:34:14.233815 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:34:14.233783 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f856065_f8f0_428e_92a6_bdd6ec801db8.slice/crio-04df069a804e351f1f66d3e030e907347250314bf02c622ac27f58891236727a WatchSource:0}: Error finding container 04df069a804e351f1f66d3e030e907347250314bf02c622ac27f58891236727a: Status 404 returned error can't find the container with id 04df069a804e351f1f66d3e030e907347250314bf02c622ac27f58891236727a Jul 29 14:34:14.257970 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.257939 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m"] Jul 29 14:34:14.260863 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:34:14.260834 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2243db38_e112_4573_b8b2_beb11309dc71.slice/crio-2ef13e74223e3df91cebe654b2dec1017acc52409e0409ba6a41a4b20bf1746f WatchSource:0}: Error finding container 2ef13e74223e3df91cebe654b2dec1017acc52409e0409ba6a41a4b20bf1746f: Status 404 returned error can't find the container with id 2ef13e74223e3df91cebe654b2dec1017acc52409e0409ba6a41a4b20bf1746f Jul 29 14:34:14.464694 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.464642 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" event={"ID":"2243db38-e112-4573-b8b2-beb11309dc71","Type":"ContainerStarted","Data":"951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc"} Jul 29 14:34:14.465003 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.464756 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" event={"ID":"2243db38-e112-4573-b8b2-beb11309dc71","Type":"ContainerStarted","Data":"2ef13e74223e3df91cebe654b2dec1017acc52409e0409ba6a41a4b20bf1746f"} Jul 29 14:34:14.465003 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.464793 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:14.466307 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.466280 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" event={"ID":"2f856065-f8f0-428e-92a6-bdd6ec801db8","Type":"ContainerStarted","Data":"613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d"} Jul 29 14:34:14.466470 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.466352 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" event={"ID":"2f856065-f8f0-428e-92a6-bdd6ec801db8","Type":"ContainerStarted","Data":"04df069a804e351f1f66d3e030e907347250314bf02c622ac27f58891236727a"} Jul 29 14:34:14.466470 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.466439 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:34:14.483475 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.483407 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" podStartSLOduration=1.483387122 podStartE2EDuration="1.483387122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:34:14.481720257 +0000 UTC m=+1980.134314106" watchObservedRunningTime="2026-07-29 14:34:14.483387122 +0000 UTC m=+1980.135980934" Jul 29 14:34:14.499489 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:14.499425 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" podStartSLOduration=1.499401674 podStartE2EDuration="1.499401674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:34:14.497670132 +0000 UTC m=+1980.150263966" watchObservedRunningTime="2026-07-29 14:34:14.499401674 +0000 UTC m=+1980.151995507" Jul 29 14:34:25.474520 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:25.474479 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:34:25.474985 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:34:25.474560 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:35:48.424092 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.424001 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m"] Jul 29 14:35:48.424764 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.424257 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" podUID="2243db38-e112-4573-b8b2-beb11309dc71" containerName="payload-processing" containerID="cri-o://951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc" gracePeriod=30 Jul 29 14:35:48.433504 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.433480 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2"] Jul 29 14:35:48.433717 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.433690 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" podUID="2f856065-f8f0-428e-92a6-bdd6ec801db8" containerName="payload-pre-processing" containerID="cri-o://613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d" gracePeriod=30 Jul 29 14:35:48.724209 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.724186 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:35:48.727325 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.727293 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:35:48.859057 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.859015 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjxbb\" (UniqueName: \"kubernetes.io/projected/2f856065-f8f0-428e-92a6-bdd6ec801db8-kube-api-access-qjxbb\") pod \"2f856065-f8f0-428e-92a6-bdd6ec801db8\" (UID: \"2f856065-f8f0-428e-92a6-bdd6ec801db8\") " Jul 29 14:35:48.859266 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.859184 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243db38-e112-4573-b8b2-beb11309dc71-plugins-config-volume\") pod \"2243db38-e112-4573-b8b2-beb11309dc71\" (UID: \"2243db38-e112-4573-b8b2-beb11309dc71\") " Jul 29 14:35:48.859463 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.859427 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgs49\" (UniqueName: \"kubernetes.io/projected/2243db38-e112-4573-b8b2-beb11309dc71-kube-api-access-kgs49\") pod \"2243db38-e112-4573-b8b2-beb11309dc71\" (UID: \"2243db38-e112-4573-b8b2-beb11309dc71\") " Jul 29 14:35:48.859559 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.859475 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f856065-f8f0-428e-92a6-bdd6ec801db8-plugins-config-volume\") pod \"2f856065-f8f0-428e-92a6-bdd6ec801db8\" (UID: \"2f856065-f8f0-428e-92a6-bdd6ec801db8\") " Jul 29 14:35:48.859635 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.859615 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2243db38-e112-4573-b8b2-beb11309dc71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2243db38-e112-4573-b8b2-beb11309dc71" (UID: "2243db38-e112-4573-b8b2-beb11309dc71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:35:48.859768 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.859749 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243db38-e112-4573-b8b2-beb11309dc71-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:35:48.859969 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.859948 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f856065-f8f0-428e-92a6-bdd6ec801db8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f856065-f8f0-428e-92a6-bdd6ec801db8" (UID: "2f856065-f8f0-428e-92a6-bdd6ec801db8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:35:48.861489 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.861467 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2243db38-e112-4573-b8b2-beb11309dc71-kube-api-access-kgs49" (OuterVolumeSpecName: "kube-api-access-kgs49") pod "2243db38-e112-4573-b8b2-beb11309dc71" (UID: "2243db38-e112-4573-b8b2-beb11309dc71"). InnerVolumeSpecName "kube-api-access-kgs49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:35:48.861685 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.861666 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f856065-f8f0-428e-92a6-bdd6ec801db8-kube-api-access-qjxbb" (OuterVolumeSpecName: "kube-api-access-qjxbb") pod "2f856065-f8f0-428e-92a6-bdd6ec801db8" (UID: "2f856065-f8f0-428e-92a6-bdd6ec801db8"). InnerVolumeSpecName "kube-api-access-qjxbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:35:48.931968 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.931931 2549 generic.go:356] "Generic (PLEG): container finished" podID="2243db38-e112-4573-b8b2-beb11309dc71" containerID="951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc" exitCode=0 Jul 29 14:35:48.932141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.931988 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" Jul 29 14:35:48.932141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.932012 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" event={"ID":"2243db38-e112-4573-b8b2-beb11309dc71","Type":"ContainerDied","Data":"951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc"} Jul 29 14:35:48.932141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.932065 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m" event={"ID":"2243db38-e112-4573-b8b2-beb11309dc71","Type":"ContainerDied","Data":"2ef13e74223e3df91cebe654b2dec1017acc52409e0409ba6a41a4b20bf1746f"} Jul 29 14:35:48.932141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.932088 2549 scope.go:117] "RemoveContainer" containerID="951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc" Jul 29 14:35:48.933517 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.933465 2549 generic.go:356] "Generic (PLEG): container finished" podID="2f856065-f8f0-428e-92a6-bdd6ec801db8" containerID="613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d" exitCode=0 Jul 29 14:35:48.933517 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.933501 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" event={"ID":"2f856065-f8f0-428e-92a6-bdd6ec801db8","Type":"ContainerDied","Data":"613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d"} Jul 29 14:35:48.933704 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.933512 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" Jul 29 14:35:48.933704 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.933546 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2" event={"ID":"2f856065-f8f0-428e-92a6-bdd6ec801db8","Type":"ContainerDied","Data":"04df069a804e351f1f66d3e030e907347250314bf02c622ac27f58891236727a"} Jul 29 14:35:48.941253 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.941222 2549 scope.go:117] "RemoveContainer" containerID="951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc" Jul 29 14:35:48.941504 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:35:48.941487 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc\": container with ID starting with 951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc not found: ID does not exist" containerID="951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc" Jul 29 14:35:48.941568 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.941511 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc"} err="failed to get container status \"951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc\": rpc error: code = NotFound desc = could not find container \"951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc\": container with ID starting with 951ea5e54c7f5d0e6afe2b9500e4f8542f9d72fd109639ab01a0f07a4a725ebc not found: ID does not exist" Jul 29 14:35:48.941568 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.941525 2549 scope.go:117] "RemoveContainer" containerID="613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d" Jul 29 14:35:48.949014 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.948999 2549 scope.go:117] "RemoveContainer" containerID="613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d" Jul 29 14:35:48.949333 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:35:48.949265 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d\": container with ID starting with 613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d not found: ID does not exist" containerID="613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d" Jul 29 14:35:48.949431 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.949342 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d"} err="failed to get container status \"613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d\": rpc error: code = NotFound desc = could not find container \"613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d\": container with ID starting with 613e0640702efa584109f36a636cb06017a7ae8c4cb5a255a6095b75448c940d not found: ID does not exist" Jul 29 14:35:48.958507 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.958484 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m"] Jul 29 14:35:48.961212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.961008 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgs49\" (UniqueName: \"kubernetes.io/projected/2243db38-e112-4573-b8b2-beb11309dc71-kube-api-access-kgs49\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:35:48.961212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.961036 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f856065-f8f0-428e-92a6-bdd6ec801db8-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:35:48.961212 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.961053 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjxbb\" (UniqueName: \"kubernetes.io/projected/2f856065-f8f0-428e-92a6-bdd6ec801db8-kube-api-access-qjxbb\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:35:48.966054 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.966033 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3fb6290-6d465f5cc4-pls2m"] Jul 29 14:35:48.976413 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.976343 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2"] Jul 29 14:35:48.978495 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:48.978475 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3fb6290-5c4bbc89d4-6s5n2"] Jul 29 14:35:50.864875 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:50.864839 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2243db38-e112-4573-b8b2-beb11309dc71" path="/var/lib/kubelet/pods/2243db38-e112-4573-b8b2-beb11309dc71/volumes" Jul 29 14:35:50.865264 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:35:50.865203 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f856065-f8f0-428e-92a6-bdd6ec801db8" path="/var/lib/kubelet/pods/2f856065-f8f0-428e-92a6-bdd6ec801db8/volumes" Jul 29 14:36:14.925607 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:36:14.925580 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:36:14.928558 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:36:14.928538 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:37:15.033204 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:15.033170 2549 scope.go:117] "RemoveContainer" containerID="68213f8bc37abdbe50d50b5f8ba795f44885fa6779255c76eea81e47570a54f6" Jul 29 14:37:15.041387 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:15.041366 2549 scope.go:117] "RemoveContainer" containerID="d3dd752f6bbf17ec8cb71302e96ef2164417d22de6b87ed76c4e68595d9528c9" Jul 29 14:37:40.238792 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.238749 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f"] Jul 29 14:37:40.239258 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.239105 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f856065-f8f0-428e-92a6-bdd6ec801db8" containerName="payload-pre-processing" Jul 29 14:37:40.239258 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.239120 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f856065-f8f0-428e-92a6-bdd6ec801db8" containerName="payload-pre-processing" Jul 29 14:37:40.239258 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.239136 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2243db38-e112-4573-b8b2-beb11309dc71" containerName="payload-processing" Jul 29 14:37:40.239258 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.239142 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2243db38-e112-4573-b8b2-beb11309dc71" containerName="payload-processing" Jul 29 14:37:40.239258 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.239198 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2243db38-e112-4573-b8b2-beb11309dc71" containerName="payload-processing" Jul 29 14:37:40.239258 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.239209 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f856065-f8f0-428e-92a6-bdd6ec801db8" containerName="payload-pre-processing" Jul 29 14:37:40.242228 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.242213 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:37:40.244909 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.244884 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-055496\"" Jul 29 14:37:40.251935 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.251914 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s"] Jul 29 14:37:40.255141 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.255120 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:40.256647 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.256625 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f"] Jul 29 14:37:40.257996 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.257978 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-055496-dockercfg-xvcq8\"" Jul 29 14:37:40.266611 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.266587 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s"] Jul 29 14:37:40.355774 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.355740 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72mz\" (UniqueName: \"kubernetes.io/projected/80147750-8635-4790-9640-7a977cc363bb-kube-api-access-v72mz\") pod \"payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f\" (UID: \"80147750-8635-4790-9640-7a977cc363bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:37:40.355947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.355848 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80147750-8635-4790-9640-7a977cc363bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f\" (UID: \"80147750-8635-4790-9640-7a977cc363bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:37:40.355947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.355908 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvjf\" (UniqueName: \"kubernetes.io/projected/3496c875-cf6d-4507-9287-620a7f321547-kube-api-access-vbvjf\") pod \"payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s\" (UID: \"3496c875-cf6d-4507-9287-620a7f321547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:40.356036 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.355979 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496c875-cf6d-4507-9287-620a7f321547-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s\" (UID: \"3496c875-cf6d-4507-9287-620a7f321547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:40.456727 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.456691 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80147750-8635-4790-9640-7a977cc363bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f\" (UID: \"80147750-8635-4790-9640-7a977cc363bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:37:40.456925 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.456752 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvjf\" (UniqueName: \"kubernetes.io/projected/3496c875-cf6d-4507-9287-620a7f321547-kube-api-access-vbvjf\") pod \"payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s\" (UID: \"3496c875-cf6d-4507-9287-620a7f321547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:40.456925 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.456804 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496c875-cf6d-4507-9287-620a7f321547-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s\" (UID: \"3496c875-cf6d-4507-9287-620a7f321547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:40.456925 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.456846 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72mz\" (UniqueName: \"kubernetes.io/projected/80147750-8635-4790-9640-7a977cc363bb-kube-api-access-v72mz\") pod \"payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f\" (UID: \"80147750-8635-4790-9640-7a977cc363bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:37:40.457500 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.457481 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80147750-8635-4790-9640-7a977cc363bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f\" (UID: \"80147750-8635-4790-9640-7a977cc363bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:37:40.457550 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.457481 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496c875-cf6d-4507-9287-620a7f321547-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s\" (UID: \"3496c875-cf6d-4507-9287-620a7f321547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:40.470048 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.470023 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvjf\" (UniqueName: \"kubernetes.io/projected/3496c875-cf6d-4507-9287-620a7f321547-kube-api-access-vbvjf\") pod \"payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s\" (UID: \"3496c875-cf6d-4507-9287-620a7f321547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:40.470158 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.470065 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72mz\" (UniqueName: \"kubernetes.io/projected/80147750-8635-4790-9640-7a977cc363bb-kube-api-access-v72mz\") pod \"payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f\" (UID: \"80147750-8635-4790-9640-7a977cc363bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:37:40.552987 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.552955 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:37:40.566179 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.566154 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:40.725728 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.725703 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s"] Jul 29 14:37:40.728099 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:37:40.728072 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3496c875_cf6d_4507_9287_620a7f321547.slice/crio-46f7fdf2c709e48dc73fa219395f3d7504b026dd3ff9d3891d188576c311056f WatchSource:0}: Error finding container 46f7fdf2c709e48dc73fa219395f3d7504b026dd3ff9d3891d188576c311056f: Status 404 returned error can't find the container with id 46f7fdf2c709e48dc73fa219395f3d7504b026dd3ff9d3891d188576c311056f Jul 29 14:37:40.901071 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:40.901047 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f"] Jul 29 14:37:40.903328 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:37:40.903287 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80147750_8635_4790_9640_7a977cc363bb.slice/crio-82a59a7cc104f3afa0fdf3399d1cb51ee957a21eba4124088d3cf914861239fa WatchSource:0}: Error finding container 82a59a7cc104f3afa0fdf3399d1cb51ee957a21eba4124088d3cf914861239fa: Status 404 returned error can't find the container with id 82a59a7cc104f3afa0fdf3399d1cb51ee957a21eba4124088d3cf914861239fa Jul 29 14:37:41.473365 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:41.473308 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" event={"ID":"3496c875-cf6d-4507-9287-620a7f321547","Type":"ContainerStarted","Data":"f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77"} Jul 29 14:37:41.473819 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:41.473387 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" event={"ID":"3496c875-cf6d-4507-9287-620a7f321547","Type":"ContainerStarted","Data":"46f7fdf2c709e48dc73fa219395f3d7504b026dd3ff9d3891d188576c311056f"} Jul 29 14:37:41.473819 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:41.473418 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:41.475072 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:41.475041 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" event={"ID":"80147750-8635-4790-9640-7a977cc363bb","Type":"ContainerStarted","Data":"c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853"} Jul 29 14:37:41.475213 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:41.475089 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" event={"ID":"80147750-8635-4790-9640-7a977cc363bb","Type":"ContainerStarted","Data":"82a59a7cc104f3afa0fdf3399d1cb51ee957a21eba4124088d3cf914861239fa"} Jul 29 14:37:41.475213 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:41.475141 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:37:41.492554 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:41.492500 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" podStartSLOduration=1.492483534 podStartE2EDuration="1.492483534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:37:41.490863401 +0000 UTC m=+2187.143457246" watchObservedRunningTime="2026-07-29 14:37:41.492483534 +0000 UTC m=+2187.145077369" Jul 29 14:37:41.506982 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:41.506942 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" podStartSLOduration=1.506927865 podStartE2EDuration="1.506927865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:37:41.506135708 +0000 UTC m=+2187.158729541" watchObservedRunningTime="2026-07-29 14:37:41.506927865 +0000 UTC m=+2187.159521736" Jul 29 14:37:52.481772 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:52.481739 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:37:52.482228 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:37:52.481812 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:38:08.762587 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.762527 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd"] Jul 29 14:38:08.766223 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.766195 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:08.769025 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.768999 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-02f676\"" Jul 29 14:38:08.777600 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.777561 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd"] Jul 29 14:38:08.800619 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.800587 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz"] Jul 29 14:38:08.804167 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.804146 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:38:08.808303 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.808276 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-02f676-dockercfg-5xdn7\"" Jul 29 14:38:08.812580 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.812549 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfwwm\" (UniqueName: \"kubernetes.io/projected/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-kube-api-access-xfwwm\") pod \"payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd\" (UID: \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:08.812705 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.812602 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd\" (UID: \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:08.816891 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.816862 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz"] Jul 29 14:38:08.913625 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.913588 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80e2c0d-face-4884-8b82-5ebede1d2a63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz\" (UID: \"b80e2c0d-face-4884-8b82-5ebede1d2a63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:38:08.913849 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.913634 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvgv6\" (UniqueName: \"kubernetes.io/projected/b80e2c0d-face-4884-8b82-5ebede1d2a63-kube-api-access-kvgv6\") pod \"payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz\" (UID: \"b80e2c0d-face-4884-8b82-5ebede1d2a63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:38:08.913849 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.913684 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfwwm\" (UniqueName: \"kubernetes.io/projected/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-kube-api-access-xfwwm\") pod \"payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd\" (UID: \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:08.913849 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.913799 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd\" (UID: \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:08.914657 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.914638 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd\" (UID: \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:08.922158 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:08.922127 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfwwm\" (UniqueName: \"kubernetes.io/projected/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-kube-api-access-xfwwm\") pod \"payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd\" (UID: \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:09.015010 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.014914 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80e2c0d-face-4884-8b82-5ebede1d2a63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz\" (UID: \"b80e2c0d-face-4884-8b82-5ebede1d2a63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:38:09.015010 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.014959 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvgv6\" (UniqueName: \"kubernetes.io/projected/b80e2c0d-face-4884-8b82-5ebede1d2a63-kube-api-access-kvgv6\") pod \"payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz\" (UID: \"b80e2c0d-face-4884-8b82-5ebede1d2a63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:38:09.015797 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.015776 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80e2c0d-face-4884-8b82-5ebede1d2a63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz\" (UID: \"b80e2c0d-face-4884-8b82-5ebede1d2a63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:38:09.023452 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.023421 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvgv6\" (UniqueName: \"kubernetes.io/projected/b80e2c0d-face-4884-8b82-5ebede1d2a63-kube-api-access-kvgv6\") pod \"payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz\" (UID: \"b80e2c0d-face-4884-8b82-5ebede1d2a63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:38:09.080072 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.080026 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:09.116104 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.116060 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:38:09.279218 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.279179 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz"] Jul 29 14:38:09.280266 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:38:09.280225 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80e2c0d_face_4884_8b82_5ebede1d2a63.slice/crio-e4b4b065acd8b4eea5dac190b33d8d5b7f7fcc525028390321dd1a1939f6dde5 WatchSource:0}: Error finding container e4b4b065acd8b4eea5dac190b33d8d5b7f7fcc525028390321dd1a1939f6dde5: Status 404 returned error can't find the container with id e4b4b065acd8b4eea5dac190b33d8d5b7f7fcc525028390321dd1a1939f6dde5 Jul 29 14:38:09.450762 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.450732 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd"] Jul 29 14:38:09.454023 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:38:09.453983 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d8115f_7d2e_4fb4_90e0_6c555a2b8a76.slice/crio-48d3d5bbb48de0d46309de2dd4e6751db2f9d937fb93f5215badfc07eb3e1abc WatchSource:0}: Error finding container 48d3d5bbb48de0d46309de2dd4e6751db2f9d937fb93f5215badfc07eb3e1abc: Status 404 returned error can't find the container with id 48d3d5bbb48de0d46309de2dd4e6751db2f9d937fb93f5215badfc07eb3e1abc Jul 29 14:38:09.617610 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.617504 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" event={"ID":"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76","Type":"ContainerStarted","Data":"9a3af3934181c8f984785f2c68f708f25870ce5fae8198c884de67dcdc64cb6b"} Jul 29 14:38:09.617610 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.617567 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" event={"ID":"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76","Type":"ContainerStarted","Data":"48d3d5bbb48de0d46309de2dd4e6751db2f9d937fb93f5215badfc07eb3e1abc"} Jul 29 14:38:09.617610 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.617595 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:09.619108 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.619082 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" event={"ID":"b80e2c0d-face-4884-8b82-5ebede1d2a63","Type":"ContainerStarted","Data":"0cac019b2086ce856b46a86b07031355c9fbe38cd3142f82bb08b0e3b9e9c22c"} Jul 29 14:38:09.619268 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.619122 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" event={"ID":"b80e2c0d-face-4884-8b82-5ebede1d2a63","Type":"ContainerStarted","Data":"e4b4b065acd8b4eea5dac190b33d8d5b7f7fcc525028390321dd1a1939f6dde5"} Jul 29 14:38:09.619268 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.619167 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:38:09.655409 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.655354 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" podStartSLOduration=1.655306402 podStartE2EDuration="1.655306402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:38:09.653981379 +0000 UTC m=+2215.306575211" watchObservedRunningTime="2026-07-29 14:38:09.655306402 +0000 UTC m=+2215.307900235" Jul 29 14:38:09.683990 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:09.683929 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" podStartSLOduration=1.6839094540000001 podStartE2EDuration="1.683909454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:38:09.682514253 +0000 UTC m=+2215.335108084" watchObservedRunningTime="2026-07-29 14:38:09.683909454 +0000 UTC m=+2215.336503288" Jul 29 14:38:20.626299 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:20.626266 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:38:20.626921 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:38:20.626371 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:40:09.623414 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.623373 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s"] Jul 29 14:40:09.625762 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.623632 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" podUID="3496c875-cf6d-4507-9287-620a7f321547" containerName="payload-processing" containerID="cri-o://f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77" gracePeriod=30 Jul 29 14:40:09.630936 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.630866 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f"] Jul 29 14:40:09.631480 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.631422 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" podUID="80147750-8635-4790-9640-7a977cc363bb" containerName="payload-pre-processing" containerID="cri-o://c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853" gracePeriod=30 Jul 29 14:40:09.921443 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.921412 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:40:09.924637 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.924608 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:40:09.976924 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.976890 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v72mz\" (UniqueName: \"kubernetes.io/projected/80147750-8635-4790-9640-7a977cc363bb-kube-api-access-v72mz\") pod \"80147750-8635-4790-9640-7a977cc363bb\" (UID: \"80147750-8635-4790-9640-7a977cc363bb\") " Jul 29 14:40:09.977126 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.976964 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbvjf\" (UniqueName: \"kubernetes.io/projected/3496c875-cf6d-4507-9287-620a7f321547-kube-api-access-vbvjf\") pod \"3496c875-cf6d-4507-9287-620a7f321547\" (UID: \"3496c875-cf6d-4507-9287-620a7f321547\") " Jul 29 14:40:09.977126 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.976991 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80147750-8635-4790-9640-7a977cc363bb-plugins-config-volume\") pod \"80147750-8635-4790-9640-7a977cc363bb\" (UID: \"80147750-8635-4790-9640-7a977cc363bb\") " Jul 29 14:40:09.977486 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.977456 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80147750-8635-4790-9640-7a977cc363bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80147750-8635-4790-9640-7a977cc363bb" (UID: "80147750-8635-4790-9640-7a977cc363bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:40:09.979290 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.979256 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3496c875-cf6d-4507-9287-620a7f321547-kube-api-access-vbvjf" (OuterVolumeSpecName: "kube-api-access-vbvjf") pod "3496c875-cf6d-4507-9287-620a7f321547" (UID: "3496c875-cf6d-4507-9287-620a7f321547"). InnerVolumeSpecName "kube-api-access-vbvjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:40:09.979426 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:09.979268 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80147750-8635-4790-9640-7a977cc363bb-kube-api-access-v72mz" (OuterVolumeSpecName: "kube-api-access-v72mz") pod "80147750-8635-4790-9640-7a977cc363bb" (UID: "80147750-8635-4790-9640-7a977cc363bb"). InnerVolumeSpecName "kube-api-access-v72mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:40:10.077632 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.077588 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496c875-cf6d-4507-9287-620a7f321547-plugins-config-volume\") pod \"3496c875-cf6d-4507-9287-620a7f321547\" (UID: \"3496c875-cf6d-4507-9287-620a7f321547\") " Jul 29 14:40:10.077852 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.077839 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80147750-8635-4790-9640-7a977cc363bb-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:40:10.077928 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.077857 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72mz\" (UniqueName: \"kubernetes.io/projected/80147750-8635-4790-9640-7a977cc363bb-kube-api-access-v72mz\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:40:10.077928 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.077871 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvjf\" (UniqueName: \"kubernetes.io/projected/3496c875-cf6d-4507-9287-620a7f321547-kube-api-access-vbvjf\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:40:10.078000 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.077959 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3496c875-cf6d-4507-9287-620a7f321547-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3496c875-cf6d-4507-9287-620a7f321547" (UID: "3496c875-cf6d-4507-9287-620a7f321547"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:40:10.178562 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.178465 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496c875-cf6d-4507-9287-620a7f321547-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:40:10.230964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.230926 2549 generic.go:356] "Generic (PLEG): container finished" podID="3496c875-cf6d-4507-9287-620a7f321547" containerID="f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77" exitCode=0 Jul 29 14:40:10.231161 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.230970 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" Jul 29 14:40:10.231161 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.231012 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" event={"ID":"3496c875-cf6d-4507-9287-620a7f321547","Type":"ContainerDied","Data":"f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77"} Jul 29 14:40:10.231161 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.231071 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s" event={"ID":"3496c875-cf6d-4507-9287-620a7f321547","Type":"ContainerDied","Data":"46f7fdf2c709e48dc73fa219395f3d7504b026dd3ff9d3891d188576c311056f"} Jul 29 14:40:10.231161 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.231090 2549 scope.go:117] "RemoveContainer" containerID="f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77" Jul 29 14:40:10.232851 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.232824 2549 generic.go:356] "Generic (PLEG): container finished" podID="80147750-8635-4790-9640-7a977cc363bb" containerID="c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853" exitCode=0 Jul 29 14:40:10.232976 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.232869 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" Jul 29 14:40:10.232976 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.232913 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" event={"ID":"80147750-8635-4790-9640-7a977cc363bb","Type":"ContainerDied","Data":"c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853"} Jul 29 14:40:10.233074 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.232973 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f" event={"ID":"80147750-8635-4790-9640-7a977cc363bb","Type":"ContainerDied","Data":"82a59a7cc104f3afa0fdf3399d1cb51ee957a21eba4124088d3cf914861239fa"} Jul 29 14:40:10.249227 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.249200 2549 scope.go:117] "RemoveContainer" containerID="f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77" Jul 29 14:40:10.249673 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:40:10.249648 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77\": container with ID starting with f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77 not found: ID does not exist" containerID="f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77" Jul 29 14:40:10.249759 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.249685 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77"} err="failed to get container status \"f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77\": rpc error: code = NotFound desc = could not find container \"f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77\": container with ID starting with f189d5b40cd13757c20a946214970e226124a026c2d3b49cfc59a622d3625d77 not found: ID does not exist" Jul 29 14:40:10.249759 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.249711 2549 scope.go:117] "RemoveContainer" containerID="c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853" Jul 29 14:40:10.259408 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.259382 2549 scope.go:117] "RemoveContainer" containerID="c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853" Jul 29 14:40:10.259772 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:40:10.259748 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853\": container with ID starting with c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853 not found: ID does not exist" containerID="c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853" Jul 29 14:40:10.259847 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.259781 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853"} err="failed to get container status \"c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853\": rpc error: code = NotFound desc = could not find container \"c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853\": container with ID starting with c47808f58ba5f384a9fed01b3b48ca2fa4c1a2e69d2f4c6a8c4e082079cf6853 not found: ID does not exist" Jul 29 14:40:10.280699 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.280666 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s"] Jul 29 14:40:10.297753 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.297714 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-055496-76bccfb94d-pw59s"] Jul 29 14:40:10.308409 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.308373 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f"] Jul 29 14:40:10.311903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.311874 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055496-74b688ff5b-w4t4f"] Jul 29 14:40:10.864487 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.864451 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3496c875-cf6d-4507-9287-620a7f321547" path="/var/lib/kubelet/pods/3496c875-cf6d-4507-9287-620a7f321547/volumes" Jul 29 14:40:10.864907 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:10.864864 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80147750-8635-4790-9640-7a977cc363bb" path="/var/lib/kubelet/pods/80147750-8635-4790-9640-7a977cc363bb/volumes" Jul 29 14:40:27.126580 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.126528 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz"] Jul 29 14:40:27.127204 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.126808 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" podUID="b80e2c0d-face-4884-8b82-5ebede1d2a63" containerName="payload-processing" containerID="cri-o://0cac019b2086ce856b46a86b07031355c9fbe38cd3142f82bb08b0e3b9e9c22c" gracePeriod=30 Jul 29 14:40:27.154479 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.143405 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd"] Jul 29 14:40:27.154479 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.143804 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" podUID="d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76" containerName="payload-pre-processing" containerID="cri-o://9a3af3934181c8f984785f2c68f708f25870ce5fae8198c884de67dcdc64cb6b" gracePeriod=30 Jul 29 14:40:27.319909 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.319856 2549 generic.go:356] "Generic (PLEG): container finished" podID="d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76" containerID="9a3af3934181c8f984785f2c68f708f25870ce5fae8198c884de67dcdc64cb6b" exitCode=0 Jul 29 14:40:27.319909 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.319899 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" event={"ID":"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76","Type":"ContainerDied","Data":"9a3af3934181c8f984785f2c68f708f25870ce5fae8198c884de67dcdc64cb6b"} Jul 29 14:40:27.322081 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.322049 2549 generic.go:356] "Generic (PLEG): container finished" podID="b80e2c0d-face-4884-8b82-5ebede1d2a63" containerID="0cac019b2086ce856b46a86b07031355c9fbe38cd3142f82bb08b0e3b9e9c22c" exitCode=0 Jul 29 14:40:27.322227 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.322122 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" event={"ID":"b80e2c0d-face-4884-8b82-5ebede1d2a63","Type":"ContainerDied","Data":"0cac019b2086ce856b46a86b07031355c9fbe38cd3142f82bb08b0e3b9e9c22c"} Jul 29 14:40:27.445084 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.445053 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:40:27.448103 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.448082 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:40:27.558497 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.558460 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfwwm\" (UniqueName: \"kubernetes.io/projected/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-kube-api-access-xfwwm\") pod \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\" (UID: \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\") " Jul 29 14:40:27.558718 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.558524 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80e2c0d-face-4884-8b82-5ebede1d2a63-plugins-config-volume\") pod \"b80e2c0d-face-4884-8b82-5ebede1d2a63\" (UID: \"b80e2c0d-face-4884-8b82-5ebede1d2a63\") " Jul 29 14:40:27.558718 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.558559 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvgv6\" (UniqueName: \"kubernetes.io/projected/b80e2c0d-face-4884-8b82-5ebede1d2a63-kube-api-access-kvgv6\") pod \"b80e2c0d-face-4884-8b82-5ebede1d2a63\" (UID: \"b80e2c0d-face-4884-8b82-5ebede1d2a63\") " Jul 29 14:40:27.558718 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.558637 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-plugins-config-volume\") pod \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\" (UID: \"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76\") " Jul 29 14:40:27.559038 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.559013 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80e2c0d-face-4884-8b82-5ebede1d2a63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b80e2c0d-face-4884-8b82-5ebede1d2a63" (UID: "b80e2c0d-face-4884-8b82-5ebede1d2a63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:40:27.559152 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.559121 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76" (UID: "d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:40:27.560797 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.560770 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-kube-api-access-xfwwm" (OuterVolumeSpecName: "kube-api-access-xfwwm") pod "d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76" (UID: "d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76"). InnerVolumeSpecName "kube-api-access-xfwwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:40:27.560890 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.560790 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80e2c0d-face-4884-8b82-5ebede1d2a63-kube-api-access-kvgv6" (OuterVolumeSpecName: "kube-api-access-kvgv6") pod "b80e2c0d-face-4884-8b82-5ebede1d2a63" (UID: "b80e2c0d-face-4884-8b82-5ebede1d2a63"). InnerVolumeSpecName "kube-api-access-kvgv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:40:27.659834 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.659787 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvgv6\" (UniqueName: \"kubernetes.io/projected/b80e2c0d-face-4884-8b82-5ebede1d2a63-kube-api-access-kvgv6\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:40:27.659834 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.659825 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:40:27.659834 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.659837 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfwwm\" (UniqueName: \"kubernetes.io/projected/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76-kube-api-access-xfwwm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:40:27.659834 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:27.659847 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80e2c0d-face-4884-8b82-5ebede1d2a63-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:40:28.330400 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.330369 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" Jul 29 14:40:28.330400 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.330390 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd" event={"ID":"d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76","Type":"ContainerDied","Data":"48d3d5bbb48de0d46309de2dd4e6751db2f9d937fb93f5215badfc07eb3e1abc"} Jul 29 14:40:28.330915 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.330463 2549 scope.go:117] "RemoveContainer" containerID="9a3af3934181c8f984785f2c68f708f25870ce5fae8198c884de67dcdc64cb6b" Jul 29 14:40:28.331937 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.331917 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" Jul 29 14:40:28.332028 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.331916 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz" event={"ID":"b80e2c0d-face-4884-8b82-5ebede1d2a63","Type":"ContainerDied","Data":"e4b4b065acd8b4eea5dac190b33d8d5b7f7fcc525028390321dd1a1939f6dde5"} Jul 29 14:40:28.341697 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.341678 2549 scope.go:117] "RemoveContainer" containerID="0cac019b2086ce856b46a86b07031355c9fbe38cd3142f82bb08b0e3b9e9c22c" Jul 29 14:40:28.359995 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.359957 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd"] Jul 29 14:40:28.365670 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.365641 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-02f676-9df5cbcfb-fzxbd"] Jul 29 14:40:28.375403 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.375370 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz"] Jul 29 14:40:28.381079 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.381047 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-02f676-6f9b95cd98-tzbvz"] Jul 29 14:40:28.865276 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.865240 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b80e2c0d-face-4884-8b82-5ebede1d2a63" path="/var/lib/kubelet/pods/b80e2c0d-face-4884-8b82-5ebede1d2a63/volumes" Jul 29 14:40:28.865671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:28.865656 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76" path="/var/lib/kubelet/pods/d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76/volumes" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.586411 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m"] Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587216 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76" containerName="payload-pre-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587237 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76" containerName="payload-pre-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587256 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b80e2c0d-face-4884-8b82-5ebede1d2a63" containerName="payload-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587265 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b80e2c0d-face-4884-8b82-5ebede1d2a63" containerName="payload-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587291 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80147750-8635-4790-9640-7a977cc363bb" containerName="payload-pre-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587301 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="80147750-8635-4790-9640-7a977cc363bb" containerName="payload-pre-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587356 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3496c875-cf6d-4507-9287-620a7f321547" containerName="payload-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587365 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3496c875-cf6d-4507-9287-620a7f321547" containerName="payload-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587537 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d8115f-7d2e-4fb4-90e0-6c555a2b8a76" containerName="payload-pre-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587552 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="80147750-8635-4790-9640-7a977cc363bb" containerName="payload-pre-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587587 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3496c875-cf6d-4507-9287-620a7f321547" containerName="payload-processing" Jul 29 14:40:39.590908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.587596 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b80e2c0d-face-4884-8b82-5ebede1d2a63" containerName="payload-processing" Jul 29 14:40:39.594185 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.594104 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:40:39.596436 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.596407 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m"] Jul 29 14:40:39.597346 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.597302 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a20e0b\"" Jul 29 14:40:39.625231 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.625183 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8"] Jul 29 14:40:39.629377 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.629347 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:39.632051 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.632021 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a20e0b-dockercfg-mvkv5\"" Jul 29 14:40:39.638739 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.638708 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8"] Jul 29 14:40:39.783985 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.783941 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c290932-660a-4780-aefa-3fa92e8a9ee6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8\" (UID: \"7c290932-660a-4780-aefa-3fa92e8a9ee6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:39.784192 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.783995 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthg6\" (UniqueName: \"kubernetes.io/projected/7c290932-660a-4780-aefa-3fa92e8a9ee6-kube-api-access-gthg6\") pod \"payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8\" (UID: \"7c290932-660a-4780-aefa-3fa92e8a9ee6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:39.784192 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.784029 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn78n\" (UniqueName: \"kubernetes.io/projected/a675925a-cd19-4cfd-ae63-65dd0e448b53-kube-api-access-zn78n\") pod \"payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m\" (UID: \"a675925a-cd19-4cfd-ae63-65dd0e448b53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:40:39.784192 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.784076 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a675925a-cd19-4cfd-ae63-65dd0e448b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m\" (UID: \"a675925a-cd19-4cfd-ae63-65dd0e448b53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:40:39.885534 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.885432 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a675925a-cd19-4cfd-ae63-65dd0e448b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m\" (UID: \"a675925a-cd19-4cfd-ae63-65dd0e448b53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:40:39.885534 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.885508 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c290932-660a-4780-aefa-3fa92e8a9ee6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8\" (UID: \"7c290932-660a-4780-aefa-3fa92e8a9ee6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:39.885751 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.885543 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthg6\" (UniqueName: \"kubernetes.io/projected/7c290932-660a-4780-aefa-3fa92e8a9ee6-kube-api-access-gthg6\") pod \"payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8\" (UID: \"7c290932-660a-4780-aefa-3fa92e8a9ee6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:39.885751 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.885565 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn78n\" (UniqueName: \"kubernetes.io/projected/a675925a-cd19-4cfd-ae63-65dd0e448b53-kube-api-access-zn78n\") pod \"payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m\" (UID: \"a675925a-cd19-4cfd-ae63-65dd0e448b53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:40:39.886203 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.886183 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a675925a-cd19-4cfd-ae63-65dd0e448b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m\" (UID: \"a675925a-cd19-4cfd-ae63-65dd0e448b53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:40:39.886281 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.886262 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c290932-660a-4780-aefa-3fa92e8a9ee6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8\" (UID: \"7c290932-660a-4780-aefa-3fa92e8a9ee6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:39.897227 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.897198 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthg6\" (UniqueName: \"kubernetes.io/projected/7c290932-660a-4780-aefa-3fa92e8a9ee6-kube-api-access-gthg6\") pod \"payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8\" (UID: \"7c290932-660a-4780-aefa-3fa92e8a9ee6\") " pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:39.897406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.897201 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn78n\" (UniqueName: \"kubernetes.io/projected/a675925a-cd19-4cfd-ae63-65dd0e448b53-kube-api-access-zn78n\") pod \"payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m\" (UID: \"a675925a-cd19-4cfd-ae63-65dd0e448b53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:40:39.906026 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.905991 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:40:39.942358 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:39.942307 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:40.062588 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.062553 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m"] Jul 29 14:40:40.065750 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:40:40.065714 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda675925a_cd19_4cfd_ae63_65dd0e448b53.slice/crio-ef4283131388570e39bc49b169c1284ca028b8d1d4f6424b7020353c497bf4f0 WatchSource:0}: Error finding container ef4283131388570e39bc49b169c1284ca028b8d1d4f6424b7020353c497bf4f0: Status 404 returned error can't find the container with id ef4283131388570e39bc49b169c1284ca028b8d1d4f6424b7020353c497bf4f0 Jul 29 14:40:40.110628 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.110595 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8"] Jul 29 14:40:40.112954 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:40:40.112925 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c290932_660a_4780_aefa_3fa92e8a9ee6.slice/crio-ad763e8b93e76331a64b982590fa49a3adc8f6fd4b39a215ef5f97b34e879307 WatchSource:0}: Error finding container ad763e8b93e76331a64b982590fa49a3adc8f6fd4b39a215ef5f97b34e879307: Status 404 returned error can't find the container with id ad763e8b93e76331a64b982590fa49a3adc8f6fd4b39a215ef5f97b34e879307 Jul 29 14:40:40.395063 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.394961 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" event={"ID":"a675925a-cd19-4cfd-ae63-65dd0e448b53","Type":"ContainerStarted","Data":"44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a"} Jul 29 14:40:40.395063 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.395029 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:40:40.395063 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.395058 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" event={"ID":"a675925a-cd19-4cfd-ae63-65dd0e448b53","Type":"ContainerStarted","Data":"ef4283131388570e39bc49b169c1284ca028b8d1d4f6424b7020353c497bf4f0"} Jul 29 14:40:40.396594 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.396562 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" event={"ID":"7c290932-660a-4780-aefa-3fa92e8a9ee6","Type":"ContainerStarted","Data":"ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8"} Jul 29 14:40:40.396734 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.396611 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" event={"ID":"7c290932-660a-4780-aefa-3fa92e8a9ee6","Type":"ContainerStarted","Data":"ad763e8b93e76331a64b982590fa49a3adc8f6fd4b39a215ef5f97b34e879307"} Jul 29 14:40:40.396734 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.396640 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:40.413168 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.413111 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" podStartSLOduration=1.41309325 podStartE2EDuration="1.41309325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:40:40.41080824 +0000 UTC m=+2366.063402098" watchObservedRunningTime="2026-07-29 14:40:40.41309325 +0000 UTC m=+2366.065687082" Jul 29 14:40:40.427897 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:40.427842 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" podStartSLOduration=1.427821275 podStartE2EDuration="1.427821275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:40:40.426547831 +0000 UTC m=+2366.079141670" watchObservedRunningTime="2026-07-29 14:40:40.427821275 +0000 UTC m=+2366.080415118" Jul 29 14:40:51.404266 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:51.404225 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:40:51.404703 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:40:51.404374 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:41:08.742252 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.742211 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r"] Jul 29 14:41:08.746034 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.746014 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:08.749527 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.749498 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ffdf30-dockercfg-gvjmd\"" Jul 29 14:41:08.749527 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.749520 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ffdf30\"" Jul 29 14:41:08.757804 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.757776 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r"] Jul 29 14:41:08.823057 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.823021 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf"] Jul 29 14:41:08.827018 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.826989 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:08.841097 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.841069 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf"] Jul 29 14:41:08.883469 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.883433 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741455be-ac2a-4e55-a4af-86e72a2f9cc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r\" (UID: \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:08.883852 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.883512 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2mh2\" (UniqueName: \"kubernetes.io/projected/741455be-ac2a-4e55-a4af-86e72a2f9cc7-kube-api-access-v2mh2\") pod \"payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r\" (UID: \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:08.984336 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.984217 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgh2r\" (UniqueName: \"kubernetes.io/projected/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-kube-api-access-cgh2r\") pod \"payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf\" (UID: \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:08.984336 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.984264 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf\" (UID: \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:08.984646 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.984360 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741455be-ac2a-4e55-a4af-86e72a2f9cc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r\" (UID: \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:08.984646 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.984506 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mh2\" (UniqueName: \"kubernetes.io/projected/741455be-ac2a-4e55-a4af-86e72a2f9cc7-kube-api-access-v2mh2\") pod \"payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r\" (UID: \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:08.985197 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.985178 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741455be-ac2a-4e55-a4af-86e72a2f9cc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r\" (UID: \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:08.994185 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:08.994106 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2mh2\" (UniqueName: \"kubernetes.io/projected/741455be-ac2a-4e55-a4af-86e72a2f9cc7-kube-api-access-v2mh2\") pod \"payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r\" (UID: \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:09.059081 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.059048 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:09.085931 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.085897 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgh2r\" (UniqueName: \"kubernetes.io/projected/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-kube-api-access-cgh2r\") pod \"payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf\" (UID: \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:09.085931 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.085932 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf\" (UID: \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:09.086633 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.086608 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf\" (UID: \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:09.095223 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.095188 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgh2r\" (UniqueName: \"kubernetes.io/projected/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-kube-api-access-cgh2r\") pod \"payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf\" (UID: \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:09.139893 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.139862 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:09.226061 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.226018 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r"] Jul 29 14:41:09.228666 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:41:09.228618 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741455be_ac2a_4e55_a4af_86e72a2f9cc7.slice/crio-0fb3d0c0fe0047051386492c5b2bf411d62cb5f9bb7e181785e01d9b30f50673 WatchSource:0}: Error finding container 0fb3d0c0fe0047051386492c5b2bf411d62cb5f9bb7e181785e01d9b30f50673: Status 404 returned error can't find the container with id 0fb3d0c0fe0047051386492c5b2bf411d62cb5f9bb7e181785e01d9b30f50673 Jul 29 14:41:09.310167 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.310141 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf"] Jul 29 14:41:09.312474 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:41:09.312429 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f0fb17_2fd0_4061_89c7_0f6dfa0113d9.slice/crio-957af8b8bed5062106c17f2f40238265977eea53f2c945e91fbef9bc2d9063e4 WatchSource:0}: Error finding container 957af8b8bed5062106c17f2f40238265977eea53f2c945e91fbef9bc2d9063e4: Status 404 returned error can't find the container with id 957af8b8bed5062106c17f2f40238265977eea53f2c945e91fbef9bc2d9063e4 Jul 29 14:41:09.544927 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.544886 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" event={"ID":"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9","Type":"ContainerStarted","Data":"84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056"} Jul 29 14:41:09.545154 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.544961 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:09.545154 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.544990 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" event={"ID":"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9","Type":"ContainerStarted","Data":"957af8b8bed5062106c17f2f40238265977eea53f2c945e91fbef9bc2d9063e4"} Jul 29 14:41:09.546787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.546758 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" event={"ID":"741455be-ac2a-4e55-a4af-86e72a2f9cc7","Type":"ContainerStarted","Data":"0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a"} Jul 29 14:41:09.546951 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.546804 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" event={"ID":"741455be-ac2a-4e55-a4af-86e72a2f9cc7","Type":"ContainerStarted","Data":"0fb3d0c0fe0047051386492c5b2bf411d62cb5f9bb7e181785e01d9b30f50673"} Jul 29 14:41:09.546951 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.546833 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:09.564257 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.564198 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" podStartSLOduration=1.5641776269999998 podStartE2EDuration="1.564177627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:41:09.562856576 +0000 UTC m=+2395.215450393" watchObservedRunningTime="2026-07-29 14:41:09.564177627 +0000 UTC m=+2395.216771460" Jul 29 14:41:09.580036 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:09.579975 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" podStartSLOduration=1.5799528600000001 podStartE2EDuration="1.57995286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:41:09.579121562 +0000 UTC m=+2395.231715395" watchObservedRunningTime="2026-07-29 14:41:09.57995286 +0000 UTC m=+2395.232546697" Jul 29 14:41:14.956767 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:14.956736 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:41:14.960384 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:14.960359 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:41:20.555851 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:20.555810 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:41:20.556288 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:20.556000 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:41:51.716625 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:51.716396 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8"] Jul 29 14:41:51.717213 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:51.716701 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" podUID="7c290932-660a-4780-aefa-3fa92e8a9ee6" containerName="payload-processing" containerID="cri-o://ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8" gracePeriod=30 Jul 29 14:41:51.723904 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:51.723865 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m"] Jul 29 14:41:51.724137 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:51.724109 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" podUID="a675925a-cd19-4cfd-ae63-65dd0e448b53" containerName="payload-pre-processing" containerID="cri-o://44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a" gracePeriod=30 Jul 29 14:41:51.989119 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:51.989089 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:41:52.003820 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.003788 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c290932-660a-4780-aefa-3fa92e8a9ee6-plugins-config-volume\") pod \"7c290932-660a-4780-aefa-3fa92e8a9ee6\" (UID: \"7c290932-660a-4780-aefa-3fa92e8a9ee6\") " Jul 29 14:41:52.003972 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.003844 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gthg6\" (UniqueName: \"kubernetes.io/projected/7c290932-660a-4780-aefa-3fa92e8a9ee6-kube-api-access-gthg6\") pod \"7c290932-660a-4780-aefa-3fa92e8a9ee6\" (UID: \"7c290932-660a-4780-aefa-3fa92e8a9ee6\") " Jul 29 14:41:52.004166 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.004141 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c290932-660a-4780-aefa-3fa92e8a9ee6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c290932-660a-4780-aefa-3fa92e8a9ee6" (UID: "7c290932-660a-4780-aefa-3fa92e8a9ee6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:41:52.006498 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.006460 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c290932-660a-4780-aefa-3fa92e8a9ee6-kube-api-access-gthg6" (OuterVolumeSpecName: "kube-api-access-gthg6") pod "7c290932-660a-4780-aefa-3fa92e8a9ee6" (UID: "7c290932-660a-4780-aefa-3fa92e8a9ee6"). InnerVolumeSpecName "kube-api-access-gthg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:41:52.023413 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.023370 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:41:52.104538 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.104497 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a675925a-cd19-4cfd-ae63-65dd0e448b53-plugins-config-volume\") pod \"a675925a-cd19-4cfd-ae63-65dd0e448b53\" (UID: \"a675925a-cd19-4cfd-ae63-65dd0e448b53\") " Jul 29 14:41:52.104538 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.104540 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn78n\" (UniqueName: \"kubernetes.io/projected/a675925a-cd19-4cfd-ae63-65dd0e448b53-kube-api-access-zn78n\") pod \"a675925a-cd19-4cfd-ae63-65dd0e448b53\" (UID: \"a675925a-cd19-4cfd-ae63-65dd0e448b53\") " Jul 29 14:41:52.104783 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.104694 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c290932-660a-4780-aefa-3fa92e8a9ee6-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:41:52.104783 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.104705 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gthg6\" (UniqueName: \"kubernetes.io/projected/7c290932-660a-4780-aefa-3fa92e8a9ee6-kube-api-access-gthg6\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:41:52.104907 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.104881 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a675925a-cd19-4cfd-ae63-65dd0e448b53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a675925a-cd19-4cfd-ae63-65dd0e448b53" (UID: "a675925a-cd19-4cfd-ae63-65dd0e448b53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:41:52.106807 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.106782 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a675925a-cd19-4cfd-ae63-65dd0e448b53-kube-api-access-zn78n" (OuterVolumeSpecName: "kube-api-access-zn78n") pod "a675925a-cd19-4cfd-ae63-65dd0e448b53" (UID: "a675925a-cd19-4cfd-ae63-65dd0e448b53"). InnerVolumeSpecName "kube-api-access-zn78n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:41:52.205246 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.205207 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a675925a-cd19-4cfd-ae63-65dd0e448b53-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:41:52.205246 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.205242 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn78n\" (UniqueName: \"kubernetes.io/projected/a675925a-cd19-4cfd-ae63-65dd0e448b53-kube-api-access-zn78n\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:41:52.765071 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.765039 2549 generic.go:356] "Generic (PLEG): container finished" podID="a675925a-cd19-4cfd-ae63-65dd0e448b53" containerID="44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a" exitCode=0 Jul 29 14:41:52.765581 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.765087 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" Jul 29 14:41:52.765581 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.765136 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" event={"ID":"a675925a-cd19-4cfd-ae63-65dd0e448b53","Type":"ContainerDied","Data":"44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a"} Jul 29 14:41:52.765581 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.765205 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m" event={"ID":"a675925a-cd19-4cfd-ae63-65dd0e448b53","Type":"ContainerDied","Data":"ef4283131388570e39bc49b169c1284ca028b8d1d4f6424b7020353c497bf4f0"} Jul 29 14:41:52.765581 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.765241 2549 scope.go:117] "RemoveContainer" containerID="44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a" Jul 29 14:41:52.766695 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.766530 2549 generic.go:356] "Generic (PLEG): container finished" podID="7c290932-660a-4780-aefa-3fa92e8a9ee6" containerID="ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8" exitCode=0 Jul 29 14:41:52.766695 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.766571 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" Jul 29 14:41:52.766695 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.766606 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" event={"ID":"7c290932-660a-4780-aefa-3fa92e8a9ee6","Type":"ContainerDied","Data":"ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8"} Jul 29 14:41:52.766695 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.766658 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8" event={"ID":"7c290932-660a-4780-aefa-3fa92e8a9ee6","Type":"ContainerDied","Data":"ad763e8b93e76331a64b982590fa49a3adc8f6fd4b39a215ef5f97b34e879307"} Jul 29 14:41:52.774990 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.774968 2549 scope.go:117] "RemoveContainer" containerID="44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a" Jul 29 14:41:52.775352 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:41:52.775331 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a\": container with ID starting with 44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a not found: ID does not exist" containerID="44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a" Jul 29 14:41:52.775424 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.775361 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a"} err="failed to get container status \"44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a\": rpc error: code = NotFound desc = could not find container \"44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a\": container with ID starting with 44fda55df81bc5a081681f0207bd7e216e6c93da2a69b7955af7c7a5c5ddc20a not found: ID does not exist" Jul 29 14:41:52.775424 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.775380 2549 scope.go:117] "RemoveContainer" containerID="ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8" Jul 29 14:41:52.783916 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.783889 2549 scope.go:117] "RemoveContainer" containerID="ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8" Jul 29 14:41:52.784225 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:41:52.784203 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8\": container with ID starting with ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8 not found: ID does not exist" containerID="ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8" Jul 29 14:41:52.784294 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.784240 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8"} err="failed to get container status \"ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8\": rpc error: code = NotFound desc = could not find container \"ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8\": container with ID starting with ddb6d0bb1b6c551c27f0efe218418df5342cdc28bd00ebf67a2e6a58ffbadaa8 not found: ID does not exist" Jul 29 14:41:52.791033 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.790999 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m"] Jul 29 14:41:52.794543 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.794517 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a20e0b-59dc99f85b-l8b2m"] Jul 29 14:41:52.809906 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.809866 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8"] Jul 29 14:41:52.813184 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.813154 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a20e0b-649767b9c6-5rrs8"] Jul 29 14:41:52.865242 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.865204 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c290932-660a-4780-aefa-3fa92e8a9ee6" path="/var/lib/kubelet/pods/7c290932-660a-4780-aefa-3fa92e8a9ee6/volumes" Jul 29 14:41:52.865621 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:41:52.865606 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a675925a-cd19-4cfd-ae63-65dd0e448b53" path="/var/lib/kubelet/pods/a675925a-cd19-4cfd-ae63-65dd0e448b53/volumes" Jul 29 14:42:14.212783 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.212745 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf"] Jul 29 14:42:14.213410 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.213012 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" podUID="c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9" containerName="payload-processing" containerID="cri-o://84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056" gracePeriod=30 Jul 29 14:42:14.220491 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.220440 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r"] Jul 29 14:42:14.220743 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.220711 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" podUID="741455be-ac2a-4e55-a4af-86e72a2f9cc7" containerName="payload-pre-processing" containerID="cri-o://0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a" gracePeriod=30 Jul 29 14:42:14.513972 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.513930 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:42:14.515949 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.515927 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:42:14.624532 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.624477 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-plugins-config-volume\") pod \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\" (UID: \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\") " Jul 29 14:42:14.624729 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.624580 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741455be-ac2a-4e55-a4af-86e72a2f9cc7-plugins-config-volume\") pod \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\" (UID: \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\") " Jul 29 14:42:14.624729 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.624699 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgh2r\" (UniqueName: \"kubernetes.io/projected/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-kube-api-access-cgh2r\") pod \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\" (UID: \"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9\") " Jul 29 14:42:14.624870 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.624820 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2mh2\" (UniqueName: \"kubernetes.io/projected/741455be-ac2a-4e55-a4af-86e72a2f9cc7-kube-api-access-v2mh2\") pod \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\" (UID: \"741455be-ac2a-4e55-a4af-86e72a2f9cc7\") " Jul 29 14:42:14.624925 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.624893 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741455be-ac2a-4e55-a4af-86e72a2f9cc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "741455be-ac2a-4e55-a4af-86e72a2f9cc7" (UID: "741455be-ac2a-4e55-a4af-86e72a2f9cc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:42:14.624975 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.624897 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9" (UID: "c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:42:14.625143 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.625125 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:42:14.625198 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.625147 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741455be-ac2a-4e55-a4af-86e72a2f9cc7-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:42:14.626987 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.626957 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-kube-api-access-cgh2r" (OuterVolumeSpecName: "kube-api-access-cgh2r") pod "c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9" (UID: "c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9"). InnerVolumeSpecName "kube-api-access-cgh2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:42:14.627125 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.627007 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741455be-ac2a-4e55-a4af-86e72a2f9cc7-kube-api-access-v2mh2" (OuterVolumeSpecName: "kube-api-access-v2mh2") pod "741455be-ac2a-4e55-a4af-86e72a2f9cc7" (UID: "741455be-ac2a-4e55-a4af-86e72a2f9cc7"). InnerVolumeSpecName "kube-api-access-v2mh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:42:14.726504 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.726456 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgh2r\" (UniqueName: \"kubernetes.io/projected/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9-kube-api-access-cgh2r\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:42:14.726504 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.726492 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2mh2\" (UniqueName: \"kubernetes.io/projected/741455be-ac2a-4e55-a4af-86e72a2f9cc7-kube-api-access-v2mh2\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:42:14.880860 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.880828 2549 generic.go:356] "Generic (PLEG): container finished" podID="c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9" containerID="84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056" exitCode=0 Jul 29 14:42:14.881047 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.880885 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" Jul 29 14:42:14.881047 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.880918 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" event={"ID":"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9","Type":"ContainerDied","Data":"84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056"} Jul 29 14:42:14.881047 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.880973 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf" event={"ID":"c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9","Type":"ContainerDied","Data":"957af8b8bed5062106c17f2f40238265977eea53f2c945e91fbef9bc2d9063e4"} Jul 29 14:42:14.881047 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.881004 2549 scope.go:117] "RemoveContainer" containerID="84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056" Jul 29 14:42:14.882475 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.882442 2549 generic.go:356] "Generic (PLEG): container finished" podID="741455be-ac2a-4e55-a4af-86e72a2f9cc7" containerID="0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a" exitCode=0 Jul 29 14:42:14.882620 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.882482 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" Jul 29 14:42:14.882620 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.882488 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" event={"ID":"741455be-ac2a-4e55-a4af-86e72a2f9cc7","Type":"ContainerDied","Data":"0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a"} Jul 29 14:42:14.882620 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.882534 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r" event={"ID":"741455be-ac2a-4e55-a4af-86e72a2f9cc7","Type":"ContainerDied","Data":"0fb3d0c0fe0047051386492c5b2bf411d62cb5f9bb7e181785e01d9b30f50673"} Jul 29 14:42:14.890462 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.890377 2549 scope.go:117] "RemoveContainer" containerID="84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056" Jul 29 14:42:14.890775 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:42:14.890755 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056\": container with ID starting with 84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056 not found: ID does not exist" containerID="84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056" Jul 29 14:42:14.890820 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.890789 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056"} err="failed to get container status \"84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056\": rpc error: code = NotFound desc = could not find container \"84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056\": container with ID starting with 84509046696fa7f1db3103732403fc745f3298f6166bfbacbfde7b142fc00056 not found: ID does not exist" Jul 29 14:42:14.890820 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.890808 2549 scope.go:117] "RemoveContainer" containerID="0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a" Jul 29 14:42:14.898441 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.898406 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r"] Jul 29 14:42:14.900407 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.900370 2549 scope.go:117] "RemoveContainer" containerID="0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a" Jul 29 14:42:14.900934 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:42:14.900910 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a\": container with ID starting with 0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a not found: ID does not exist" containerID="0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a" Jul 29 14:42:14.901056 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.900940 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a"} err="failed to get container status \"0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a\": rpc error: code = NotFound desc = could not find container \"0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a\": container with ID starting with 0152e82c3c58e15317b7c81663fe08702ddf27b1fdd1017219e75f443a88e10a not found: ID does not exist" Jul 29 14:42:14.902463 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.902424 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ffdf30-db88f6884-fl85r"] Jul 29 14:42:14.913393 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.913349 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf"] Jul 29 14:42:14.916448 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:14.916414 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdf30-7648ffd956-466hf"] Jul 29 14:42:16.864901 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:16.864860 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="741455be-ac2a-4e55-a4af-86e72a2f9cc7" path="/var/lib/kubelet/pods/741455be-ac2a-4e55-a4af-86e72a2f9cc7/volumes" Jul 29 14:42:16.865332 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:16.865257 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9" path="/var/lib/kubelet/pods/c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9/volumes" Jul 29 14:42:26.273102 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273063 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8"] Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273464 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741455be-ac2a-4e55-a4af-86e72a2f9cc7" containerName="payload-pre-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273477 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="741455be-ac2a-4e55-a4af-86e72a2f9cc7" containerName="payload-pre-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273489 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c290932-660a-4780-aefa-3fa92e8a9ee6" containerName="payload-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273496 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c290932-660a-4780-aefa-3fa92e8a9ee6" containerName="payload-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273514 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a675925a-cd19-4cfd-ae63-65dd0e448b53" containerName="payload-pre-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273522 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a675925a-cd19-4cfd-ae63-65dd0e448b53" containerName="payload-pre-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273531 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9" containerName="payload-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273536 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9" containerName="payload-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273595 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a675925a-cd19-4cfd-ae63-65dd0e448b53" containerName="payload-pre-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273605 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c290932-660a-4780-aefa-3fa92e8a9ee6" containerName="payload-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273611 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f0fb17-2fd0-4061-89c7-0f6dfa0113d9" containerName="payload-processing" Jul 29 14:42:26.273678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.273620 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="741455be-ac2a-4e55-a4af-86e72a2f9cc7" containerName="payload-pre-processing" Jul 29 14:42:26.276694 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.276660 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:26.280276 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.280248 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-61bc0a\"" Jul 29 14:42:26.280456 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.280283 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-61bc0a-dockercfg-9tlvm\"" Jul 29 14:42:26.284429 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.284402 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk"] Jul 29 14:42:26.288665 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.288632 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8"] Jul 29 14:42:26.288822 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.288770 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:26.302656 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.302610 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk"] Jul 29 14:42:26.433134 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.433093 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-422nh\" (UniqueName: \"kubernetes.io/projected/15f5659a-98b6-444e-a4eb-5ace33fd7b56-kube-api-access-422nh\") pod \"payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8\" (UID: \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:26.433363 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.433141 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5659a-98b6-444e-a4eb-5ace33fd7b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8\" (UID: \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:26.433363 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.433228 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whpx\" (UniqueName: \"kubernetes.io/projected/303318c6-72ca-4a8e-b55f-8e46eabf875c-kube-api-access-5whpx\") pod \"payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk\" (UID: \"303318c6-72ca-4a8e-b55f-8e46eabf875c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:26.433486 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.433417 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303318c6-72ca-4a8e-b55f-8e46eabf875c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk\" (UID: \"303318c6-72ca-4a8e-b55f-8e46eabf875c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:26.534238 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.534129 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303318c6-72ca-4a8e-b55f-8e46eabf875c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk\" (UID: \"303318c6-72ca-4a8e-b55f-8e46eabf875c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:26.534238 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.534182 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-422nh\" (UniqueName: \"kubernetes.io/projected/15f5659a-98b6-444e-a4eb-5ace33fd7b56-kube-api-access-422nh\") pod \"payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8\" (UID: \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:26.534238 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.534199 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5659a-98b6-444e-a4eb-5ace33fd7b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8\" (UID: \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:26.534238 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.534225 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whpx\" (UniqueName: \"kubernetes.io/projected/303318c6-72ca-4a8e-b55f-8e46eabf875c-kube-api-access-5whpx\") pod \"payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk\" (UID: \"303318c6-72ca-4a8e-b55f-8e46eabf875c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:26.535037 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.535011 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5659a-98b6-444e-a4eb-5ace33fd7b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8\" (UID: \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:26.535124 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.535011 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303318c6-72ca-4a8e-b55f-8e46eabf875c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk\" (UID: \"303318c6-72ca-4a8e-b55f-8e46eabf875c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:26.542756 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.542726 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whpx\" (UniqueName: \"kubernetes.io/projected/303318c6-72ca-4a8e-b55f-8e46eabf875c-kube-api-access-5whpx\") pod \"payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk\" (UID: \"303318c6-72ca-4a8e-b55f-8e46eabf875c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:26.542908 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.542783 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-422nh\" (UniqueName: \"kubernetes.io/projected/15f5659a-98b6-444e-a4eb-5ace33fd7b56-kube-api-access-422nh\") pod \"payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8\" (UID: \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:26.590811 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.590771 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:26.602153 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.602117 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:26.770991 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.770905 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8"] Jul 29 14:42:26.773448 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:42:26.773411 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f5659a_98b6_444e_a4eb_5ace33fd7b56.slice/crio-99be740d83afc1e0b33da0553f47157c7d1c63cf5ac61c2cfd70716e945b0f88 WatchSource:0}: Error finding container 99be740d83afc1e0b33da0553f47157c7d1c63cf5ac61c2cfd70716e945b0f88: Status 404 returned error can't find the container with id 99be740d83afc1e0b33da0553f47157c7d1c63cf5ac61c2cfd70716e945b0f88 Jul 29 14:42:26.803069 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.803040 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk"] Jul 29 14:42:26.812812 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:42:26.812775 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303318c6_72ca_4a8e_b55f_8e46eabf875c.slice/crio-2922dea64eff2045c5b4aca681417babb54d04699768588ff5d6dfa762600b47 WatchSource:0}: Error finding container 2922dea64eff2045c5b4aca681417babb54d04699768588ff5d6dfa762600b47: Status 404 returned error can't find the container with id 2922dea64eff2045c5b4aca681417babb54d04699768588ff5d6dfa762600b47 Jul 29 14:42:26.945834 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.945789 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" event={"ID":"303318c6-72ca-4a8e-b55f-8e46eabf875c","Type":"ContainerStarted","Data":"75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb"} Jul 29 14:42:26.946049 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.945867 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" event={"ID":"303318c6-72ca-4a8e-b55f-8e46eabf875c","Type":"ContainerStarted","Data":"2922dea64eff2045c5b4aca681417babb54d04699768588ff5d6dfa762600b47"} Jul 29 14:42:26.946166 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.946135 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:26.948691 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.948656 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" event={"ID":"15f5659a-98b6-444e-a4eb-5ace33fd7b56","Type":"ContainerStarted","Data":"900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860"} Jul 29 14:42:26.948890 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.948715 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" event={"ID":"15f5659a-98b6-444e-a4eb-5ace33fd7b56","Type":"ContainerStarted","Data":"99be740d83afc1e0b33da0553f47157c7d1c63cf5ac61c2cfd70716e945b0f88"} Jul 29 14:42:26.948890 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.948837 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:26.963993 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.963940 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" podStartSLOduration=0.963923413 podStartE2EDuration="963.923413ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:42:26.963133661 +0000 UTC m=+2472.615727503" watchObservedRunningTime="2026-07-29 14:42:26.963923413 +0000 UTC m=+2472.616517224" Jul 29 14:42:26.979565 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:26.979485 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" podStartSLOduration=0.979458647 podStartE2EDuration="979.458647ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:42:26.978814036 +0000 UTC m=+2472.631407870" watchObservedRunningTime="2026-07-29 14:42:26.979458647 +0000 UTC m=+2472.632052488" Jul 29 14:42:37.955555 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:37.955498 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:42:37.956065 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:37.955643 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:42:54.967744 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:54.967702 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz"] Jul 29 14:42:54.971596 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:54.971569 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:42:54.974214 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:54.974187 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-532c20\"" Jul 29 14:42:54.981901 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:54.981805 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn"] Jul 29 14:42:54.985995 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:54.985964 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz"] Jul 29 14:42:54.986161 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:54.986089 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:42:54.989468 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:54.989436 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-532c20-dockercfg-6wkg6\"" Jul 29 14:42:54.998607 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:54.998571 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn"] Jul 29 14:42:55.105971 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.105930 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz\" (UID: \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:42:55.106184 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.105983 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbzcn\" (UniqueName: \"kubernetes.io/projected/8135473d-43a2-4699-9729-66204f3a9138-kube-api-access-tbzcn\") pod \"payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn\" (UID: \"8135473d-43a2-4699-9729-66204f3a9138\") " pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:42:55.106184 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.106020 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97w4w\" (UniqueName: \"kubernetes.io/projected/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-kube-api-access-97w4w\") pod \"payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz\" (UID: \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:42:55.106184 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.106043 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8135473d-43a2-4699-9729-66204f3a9138-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn\" (UID: \"8135473d-43a2-4699-9729-66204f3a9138\") " pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:42:55.207569 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.207497 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97w4w\" (UniqueName: \"kubernetes.io/projected/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-kube-api-access-97w4w\") pod \"payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz\" (UID: \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:42:55.207569 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.207567 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8135473d-43a2-4699-9729-66204f3a9138-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn\" (UID: \"8135473d-43a2-4699-9729-66204f3a9138\") " pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:42:55.207822 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.207667 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz\" (UID: \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:42:55.207822 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.207694 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbzcn\" (UniqueName: \"kubernetes.io/projected/8135473d-43a2-4699-9729-66204f3a9138-kube-api-access-tbzcn\") pod \"payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn\" (UID: \"8135473d-43a2-4699-9729-66204f3a9138\") " pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:42:55.208526 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.208490 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8135473d-43a2-4699-9729-66204f3a9138-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn\" (UID: \"8135473d-43a2-4699-9729-66204f3a9138\") " pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:42:55.208526 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.208521 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz\" (UID: \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:42:55.216574 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.216535 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97w4w\" (UniqueName: \"kubernetes.io/projected/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-kube-api-access-97w4w\") pod \"payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz\" (UID: \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:42:55.216574 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.216553 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbzcn\" (UniqueName: \"kubernetes.io/projected/8135473d-43a2-4699-9729-66204f3a9138-kube-api-access-tbzcn\") pod \"payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn\" (UID: \"8135473d-43a2-4699-9729-66204f3a9138\") " pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:42:55.286577 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.286541 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:42:55.303203 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.303130 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:42:55.449418 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.449306 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz"] Jul 29 14:42:55.452804 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:42:55.452768 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d093f2_69f9_4eed_a1b5_9a0da8f30fd8.slice/crio-3db33c36954de953703c9bf3e4d3e7534518235aab4514276bdf5e3b77f90d40 WatchSource:0}: Error finding container 3db33c36954de953703c9bf3e4d3e7534518235aab4514276bdf5e3b77f90d40: Status 404 returned error can't find the container with id 3db33c36954de953703c9bf3e4d3e7534518235aab4514276bdf5e3b77f90d40 Jul 29 14:42:55.480948 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:55.480894 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn"] Jul 29 14:42:55.485635 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:42:55.485602 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8135473d_43a2_4699_9729_66204f3a9138.slice/crio-b309efadafee4f0f87209ec6355b7a02404a5fe093c2e05919373f21ba47a5af WatchSource:0}: Error finding container b309efadafee4f0f87209ec6355b7a02404a5fe093c2e05919373f21ba47a5af: Status 404 returned error can't find the container with id b309efadafee4f0f87209ec6355b7a02404a5fe093c2e05919373f21ba47a5af Jul 29 14:42:56.096793 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:56.096750 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" event={"ID":"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8","Type":"ContainerStarted","Data":"9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4"} Jul 29 14:42:56.097277 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:56.096817 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" event={"ID":"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8","Type":"ContainerStarted","Data":"3db33c36954de953703c9bf3e4d3e7534518235aab4514276bdf5e3b77f90d40"} Jul 29 14:42:56.097277 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:56.096947 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:42:56.098511 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:56.098484 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" event={"ID":"8135473d-43a2-4699-9729-66204f3a9138","Type":"ContainerStarted","Data":"8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea"} Jul 29 14:42:56.098615 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:56.098528 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" event={"ID":"8135473d-43a2-4699-9729-66204f3a9138","Type":"ContainerStarted","Data":"b309efadafee4f0f87209ec6355b7a02404a5fe093c2e05919373f21ba47a5af"} Jul 29 14:42:56.098615 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:56.098548 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:42:56.114686 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:56.114629 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" podStartSLOduration=2.114609464 podStartE2EDuration="2.114609464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:42:56.112940091 +0000 UTC m=+2501.765533925" watchObservedRunningTime="2026-07-29 14:42:56.114609464 +0000 UTC m=+2501.767203298" Jul 29 14:42:56.128246 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:42:56.128181 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" podStartSLOduration=2.128159088 podStartE2EDuration="2.128159088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:42:56.127373732 +0000 UTC m=+2501.779967567" watchObservedRunningTime="2026-07-29 14:42:56.128159088 +0000 UTC m=+2501.780752973" Jul 29 14:43:07.106727 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:43:07.106646 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:43:07.106727 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:43:07.106721 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:46:14.985871 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:46:14.985836 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:46:14.990996 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:46:14.990965 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:51:15.014440 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:51:15.014409 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:51:15.020525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:51:15.020500 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:53:09.075626 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.075585 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr"] Jul 29 14:53:09.079149 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.079130 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:09.082014 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.081989 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-19afc1\"" Jul 29 14:53:09.082014 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.082007 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-19afc1-dockercfg-ccnj4\"" Jul 29 14:53:09.089553 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.089522 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr"] Jul 29 14:53:09.102089 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.102053 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg8r8\" (UniqueName: \"kubernetes.io/projected/a5bd7ec1-eb80-43e5-9b78-546326d6656d-kube-api-access-lg8r8\") pod \"payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr\" (UID: \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:09.102306 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.102115 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd7ec1-eb80-43e5-9b78-546326d6656d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr\" (UID: \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:09.188358 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.188301 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd"] Jul 29 14:53:09.192053 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.192028 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:09.203189 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.203155 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd7ec1-eb80-43e5-9b78-546326d6656d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr\" (UID: \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:09.203392 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.203283 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd\" (UID: \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:09.203392 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.203341 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg8r8\" (UniqueName: \"kubernetes.io/projected/a5bd7ec1-eb80-43e5-9b78-546326d6656d-kube-api-access-lg8r8\") pod \"payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr\" (UID: \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:09.203392 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.203363 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggm79\" (UniqueName: \"kubernetes.io/projected/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-kube-api-access-ggm79\") pod \"payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd\" (UID: \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:09.204004 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.203983 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd7ec1-eb80-43e5-9b78-546326d6656d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr\" (UID: \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:09.208627 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.208591 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd"] Jul 29 14:53:09.218990 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.218954 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg8r8\" (UniqueName: \"kubernetes.io/projected/a5bd7ec1-eb80-43e5-9b78-546326d6656d-kube-api-access-lg8r8\") pod \"payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr\" (UID: \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:09.304096 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.304049 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd\" (UID: \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:09.304096 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.304102 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggm79\" (UniqueName: \"kubernetes.io/projected/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-kube-api-access-ggm79\") pod \"payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd\" (UID: \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:09.305108 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.305079 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd\" (UID: \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:09.312964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.312935 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggm79\" (UniqueName: \"kubernetes.io/projected/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-kube-api-access-ggm79\") pod \"payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd\" (UID: \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:09.391962 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.391852 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:09.504516 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.504454 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:09.552140 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.552101 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr"] Jul 29 14:53:09.556286 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:53:09.556247 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bd7ec1_eb80_43e5_9b78_546326d6656d.slice/crio-efacb4ed44c360e095e6f978dcdbaa78c756204398c9207c9b7d99cbea9dd65d WatchSource:0}: Error finding container efacb4ed44c360e095e6f978dcdbaa78c756204398c9207c9b7d99cbea9dd65d: Status 404 returned error can't find the container with id efacb4ed44c360e095e6f978dcdbaa78c756204398c9207c9b7d99cbea9dd65d Jul 29 14:53:09.667153 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:09.667121 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd"] Jul 29 14:53:09.670476 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:53:09.670164 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88f6f4a_b78a_4a63_bb28_92c3d6635d76.slice/crio-1c4279d33194739507fd8e0117bd8d51681d0cf8a9595f089f950195b43dd365 WatchSource:0}: Error finding container 1c4279d33194739507fd8e0117bd8d51681d0cf8a9595f089f950195b43dd365: Status 404 returned error can't find the container with id 1c4279d33194739507fd8e0117bd8d51681d0cf8a9595f089f950195b43dd365 Jul 29 14:53:10.159525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:10.159478 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" event={"ID":"e88f6f4a-b78a-4a63-bb28-92c3d6635d76","Type":"ContainerStarted","Data":"30649a526be22382ec999e6da0d400aad308b2a2d2ba8aa56dc4b45c653e706c"} Jul 29 14:53:10.160052 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:10.159568 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" event={"ID":"e88f6f4a-b78a-4a63-bb28-92c3d6635d76","Type":"ContainerStarted","Data":"1c4279d33194739507fd8e0117bd8d51681d0cf8a9595f089f950195b43dd365"} Jul 29 14:53:10.160052 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:10.159689 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:10.161371 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:10.161342 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" event={"ID":"a5bd7ec1-eb80-43e5-9b78-546326d6656d","Type":"ContainerStarted","Data":"2e6ed5b8d52c0ba4341212017234edc7981024644d97047baa271e980038c74b"} Jul 29 14:53:10.161506 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:10.161388 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" event={"ID":"a5bd7ec1-eb80-43e5-9b78-546326d6656d","Type":"ContainerStarted","Data":"efacb4ed44c360e095e6f978dcdbaa78c756204398c9207c9b7d99cbea9dd65d"} Jul 29 14:53:10.161506 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:10.161416 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:10.179684 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:10.179614 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" podStartSLOduration=1.179587696 podStartE2EDuration="1.179587696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:53:10.176192789 +0000 UTC m=+3115.828786623" watchObservedRunningTime="2026-07-29 14:53:10.179587696 +0000 UTC m=+3115.832181533" Jul 29 14:53:10.198009 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:10.197951 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" podStartSLOduration=1.197929337 podStartE2EDuration="1.197929337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:53:10.196138662 +0000 UTC m=+3115.848732492" watchObservedRunningTime="2026-07-29 14:53:10.197929337 +0000 UTC m=+3115.850523162" Jul 29 14:53:21.169550 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:21.169513 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:53:21.169983 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:21.169594 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:53:37.905147 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:37.905057 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t"] Jul 29 14:53:37.908703 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:37.908679 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:53:37.911896 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:37.911875 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-726aca\"" Jul 29 14:53:37.929343 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:37.929291 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t"] Jul 29 14:53:37.943672 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:37.943632 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x"] Jul 29 14:53:37.948555 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:37.948495 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:37.951534 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:37.951511 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-726aca-dockercfg-4plq8\"" Jul 29 14:53:37.958230 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:37.958200 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x"] Jul 29 14:53:38.101805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.101763 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hl9\" (UniqueName: \"kubernetes.io/projected/d4268033-0b47-40bb-b553-3a0d7a25c38b-kube-api-access-p6hl9\") pod \"payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x\" (UID: \"d4268033-0b47-40bb-b553-3a0d7a25c38b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:38.101805 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.101815 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8gc\" (UniqueName: \"kubernetes.io/projected/35eaf4a4-ac5a-411e-b518-83bffd851a22-kube-api-access-nj8gc\") pod \"payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t\" (UID: \"35eaf4a4-ac5a-411e-b518-83bffd851a22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:53:38.102067 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.101832 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4268033-0b47-40bb-b553-3a0d7a25c38b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x\" (UID: \"d4268033-0b47-40bb-b553-3a0d7a25c38b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:38.102067 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.101976 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eaf4a4-ac5a-411e-b518-83bffd851a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t\" (UID: \"35eaf4a4-ac5a-411e-b518-83bffd851a22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:53:38.203166 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.203069 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hl9\" (UniqueName: \"kubernetes.io/projected/d4268033-0b47-40bb-b553-3a0d7a25c38b-kube-api-access-p6hl9\") pod \"payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x\" (UID: \"d4268033-0b47-40bb-b553-3a0d7a25c38b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:38.203166 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.203116 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8gc\" (UniqueName: \"kubernetes.io/projected/35eaf4a4-ac5a-411e-b518-83bffd851a22-kube-api-access-nj8gc\") pod \"payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t\" (UID: \"35eaf4a4-ac5a-411e-b518-83bffd851a22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:53:38.203166 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.203136 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4268033-0b47-40bb-b553-3a0d7a25c38b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x\" (UID: \"d4268033-0b47-40bb-b553-3a0d7a25c38b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:38.203468 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.203207 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eaf4a4-ac5a-411e-b518-83bffd851a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t\" (UID: \"35eaf4a4-ac5a-411e-b518-83bffd851a22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:53:38.204057 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.204035 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eaf4a4-ac5a-411e-b518-83bffd851a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t\" (UID: \"35eaf4a4-ac5a-411e-b518-83bffd851a22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:53:38.204117 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.204036 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4268033-0b47-40bb-b553-3a0d7a25c38b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x\" (UID: \"d4268033-0b47-40bb-b553-3a0d7a25c38b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:38.221023 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.220986 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hl9\" (UniqueName: \"kubernetes.io/projected/d4268033-0b47-40bb-b553-3a0d7a25c38b-kube-api-access-p6hl9\") pod \"payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x\" (UID: \"d4268033-0b47-40bb-b553-3a0d7a25c38b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:38.240537 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.240501 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8gc\" (UniqueName: \"kubernetes.io/projected/35eaf4a4-ac5a-411e-b518-83bffd851a22-kube-api-access-nj8gc\") pod \"payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t\" (UID: \"35eaf4a4-ac5a-411e-b518-83bffd851a22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:53:38.266177 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.266135 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:38.421837 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.421807 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x"] Jul 29 14:53:38.424238 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:53:38.424205 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4268033_0b47_40bb_b553_3a0d7a25c38b.slice/crio-091f2bda6004a6fb545015b4e6f1e017e71ec5d8f7543b380a01ff04507283ef WatchSource:0}: Error finding container 091f2bda6004a6fb545015b4e6f1e017e71ec5d8f7543b380a01ff04507283ef: Status 404 returned error can't find the container with id 091f2bda6004a6fb545015b4e6f1e017e71ec5d8f7543b380a01ff04507283ef Jul 29 14:53:38.520080 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.520029 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:53:38.674737 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:38.674700 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t"] Jul 29 14:53:38.676829 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:53:38.676790 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35eaf4a4_ac5a_411e_b518_83bffd851a22.slice/crio-b78d1852363ca9b4d5ed28072f37c3889ca595c9c8b2cd0c17f2dea1321fea0b WatchSource:0}: Error finding container b78d1852363ca9b4d5ed28072f37c3889ca595c9c8b2cd0c17f2dea1321fea0b: Status 404 returned error can't find the container with id b78d1852363ca9b4d5ed28072f37c3889ca595c9c8b2cd0c17f2dea1321fea0b Jul 29 14:53:39.317143 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:39.317094 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" event={"ID":"35eaf4a4-ac5a-411e-b518-83bffd851a22","Type":"ContainerStarted","Data":"ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7"} Jul 29 14:53:39.317710 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:39.317163 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" event={"ID":"35eaf4a4-ac5a-411e-b518-83bffd851a22","Type":"ContainerStarted","Data":"b78d1852363ca9b4d5ed28072f37c3889ca595c9c8b2cd0c17f2dea1321fea0b"} Jul 29 14:53:39.317710 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:39.317192 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:53:39.319173 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:39.319140 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" event={"ID":"d4268033-0b47-40bb-b553-3a0d7a25c38b","Type":"ContainerStarted","Data":"2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156"} Jul 29 14:53:39.319372 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:39.319200 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" event={"ID":"d4268033-0b47-40bb-b553-3a0d7a25c38b","Type":"ContainerStarted","Data":"091f2bda6004a6fb545015b4e6f1e017e71ec5d8f7543b380a01ff04507283ef"} Jul 29 14:53:39.319372 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:39.319259 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:39.338985 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:39.338916 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" podStartSLOduration=2.338891441 podStartE2EDuration="2.338891441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:53:39.338540141 +0000 UTC m=+3144.991133976" watchObservedRunningTime="2026-07-29 14:53:39.338891441 +0000 UTC m=+3144.991485276" Jul 29 14:53:39.368503 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:39.368435 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" podStartSLOduration=2.368411498 podStartE2EDuration="2.368411498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:53:39.365550892 +0000 UTC m=+3145.018144726" watchObservedRunningTime="2026-07-29 14:53:39.368411498 +0000 UTC m=+3145.021005333" Jul 29 14:53:50.325820 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:50.325785 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:53:50.326237 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:53:50.325884 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:54:08.965878 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:08.965849 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qswsb_0b59c1bd-570e-455f-89d7-d9e7a2856d63/payload-processing/0.log" Jul 29 14:54:09.095839 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:09.095811 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x_d4268033-0b47-40bb-b553-3a0d7a25c38b/payload-processing/0.log" Jul 29 14:54:53.922103 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:53.922068 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr_a5bd7ec1-eb80-43e5-9b78-546326d6656d/payload-processing/0.log" Jul 29 14:54:54.054953 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:54.054916 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qswsb_0b59c1bd-570e-455f-89d7-d9e7a2856d63/payload-processing/0.log" Jul 29 14:54:54.176424 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:54.176308 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x_d4268033-0b47-40bb-b553-3a0d7a25c38b/payload-processing/0.log" Jul 29 14:54:59.293521 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.293477 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c"] Jul 29 14:54:59.296903 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.296877 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:54:59.299992 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.299969 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ad0bed\"" Jul 29 14:54:59.300139 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.300037 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ad0bed-dockercfg-rx7v5\"" Jul 29 14:54:59.308113 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.308081 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac754512-9c9d-47ed-a15e-14f4450e8c00-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c\" (UID: \"ac754512-9c9d-47ed-a15e-14f4450e8c00\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:54:59.308288 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.308123 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvws\" (UniqueName: \"kubernetes.io/projected/ac754512-9c9d-47ed-a15e-14f4450e8c00-kube-api-access-wkvws\") pod \"payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c\" (UID: \"ac754512-9c9d-47ed-a15e-14f4450e8c00\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:54:59.309696 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.309666 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c"] Jul 29 14:54:59.313096 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.313070 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl"] Jul 29 14:54:59.317270 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.317247 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:54:59.331731 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.331697 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl"] Jul 29 14:54:59.408639 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.408602 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqn4\" (UniqueName: \"kubernetes.io/projected/2f4b781d-77be-4920-a063-30cfa173cc94-kube-api-access-nkqn4\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl\" (UID: \"2f4b781d-77be-4920-a063-30cfa173cc94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:54:59.408820 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.408650 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac754512-9c9d-47ed-a15e-14f4450e8c00-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c\" (UID: \"ac754512-9c9d-47ed-a15e-14f4450e8c00\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:54:59.408820 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.408709 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvws\" (UniqueName: \"kubernetes.io/projected/ac754512-9c9d-47ed-a15e-14f4450e8c00-kube-api-access-wkvws\") pod \"payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c\" (UID: \"ac754512-9c9d-47ed-a15e-14f4450e8c00\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:54:59.408820 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.408806 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4b781d-77be-4920-a063-30cfa173cc94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl\" (UID: \"2f4b781d-77be-4920-a063-30cfa173cc94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:54:59.409525 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.409498 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac754512-9c9d-47ed-a15e-14f4450e8c00-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c\" (UID: \"ac754512-9c9d-47ed-a15e-14f4450e8c00\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:54:59.417794 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.417746 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvws\" (UniqueName: \"kubernetes.io/projected/ac754512-9c9d-47ed-a15e-14f4450e8c00-kube-api-access-wkvws\") pod \"payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c\" (UID: \"ac754512-9c9d-47ed-a15e-14f4450e8c00\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:54:59.510002 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.509963 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqn4\" (UniqueName: \"kubernetes.io/projected/2f4b781d-77be-4920-a063-30cfa173cc94-kube-api-access-nkqn4\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl\" (UID: \"2f4b781d-77be-4920-a063-30cfa173cc94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:54:59.510185 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.510061 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4b781d-77be-4920-a063-30cfa173cc94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl\" (UID: \"2f4b781d-77be-4920-a063-30cfa173cc94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:54:59.510767 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.510748 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4b781d-77be-4920-a063-30cfa173cc94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl\" (UID: \"2f4b781d-77be-4920-a063-30cfa173cc94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:54:59.526896 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.526865 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqn4\" (UniqueName: \"kubernetes.io/projected/2f4b781d-77be-4920-a063-30cfa173cc94-kube-api-access-nkqn4\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl\" (UID: \"2f4b781d-77be-4920-a063-30cfa173cc94\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:54:59.609888 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.609699 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:54:59.630202 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.630168 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:54:59.772199 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.772158 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c"] Jul 29 14:54:59.782028 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:54:59.781989 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac754512_9c9d_47ed_a15e_14f4450e8c00.slice/crio-7beb95633b9f9b6d294c38f21ed20775fcce35242e87c69072aba16f180356d7 WatchSource:0}: Error finding container 7beb95633b9f9b6d294c38f21ed20775fcce35242e87c69072aba16f180356d7: Status 404 returned error can't find the container with id 7beb95633b9f9b6d294c38f21ed20775fcce35242e87c69072aba16f180356d7 Jul 29 14:54:59.808850 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:54:59.808793 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl"] Jul 29 14:54:59.809164 ip-10-0-130-28 kubenswrapper[2549]: W0729 14:54:59.809140 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4b781d_77be_4920_a063_30cfa173cc94.slice/crio-d097c4f5483228d901361648bb3d6b8fe739ade93b182c7520475518fea7aa78 WatchSource:0}: Error finding container d097c4f5483228d901361648bb3d6b8fe739ade93b182c7520475518fea7aa78: Status 404 returned error can't find the container with id d097c4f5483228d901361648bb3d6b8fe739ade93b182c7520475518fea7aa78 Jul 29 14:55:00.739130 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:00.739089 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" event={"ID":"2f4b781d-77be-4920-a063-30cfa173cc94","Type":"ContainerStarted","Data":"abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3"} Jul 29 14:55:00.739678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:00.739163 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" event={"ID":"2f4b781d-77be-4920-a063-30cfa173cc94","Type":"ContainerStarted","Data":"d097c4f5483228d901361648bb3d6b8fe739ade93b182c7520475518fea7aa78"} Jul 29 14:55:00.739678 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:00.739326 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:55:00.741081 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:00.741057 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" event={"ID":"ac754512-9c9d-47ed-a15e-14f4450e8c00","Type":"ContainerStarted","Data":"7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df"} Jul 29 14:55:00.741180 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:00.741099 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" event={"ID":"ac754512-9c9d-47ed-a15e-14f4450e8c00","Type":"ContainerStarted","Data":"7beb95633b9f9b6d294c38f21ed20775fcce35242e87c69072aba16f180356d7"} Jul 29 14:55:00.741180 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:00.741118 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:55:00.757498 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:00.757441 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" podStartSLOduration=1.75742391 podStartE2EDuration="1.75742391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:55:00.75510984 +0000 UTC m=+3226.407703684" watchObservedRunningTime="2026-07-29 14:55:00.75742391 +0000 UTC m=+3226.410017721" Jul 29 14:55:00.772905 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:00.772850 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" podStartSLOduration=1.772821388 podStartE2EDuration="1.772821388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:55:00.77203275 +0000 UTC m=+3226.424626597" watchObservedRunningTime="2026-07-29 14:55:00.772821388 +0000 UTC m=+3226.425415221" Jul 29 14:55:11.748153 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:11.748070 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:55:11.748153 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:11.748153 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:55:29.426223 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.426190 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c"] Jul 29 14:55:29.426845 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.426436 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" podUID="ac754512-9c9d-47ed-a15e-14f4450e8c00" containerName="payload-processing" containerID="cri-o://7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df" gracePeriod=30 Jul 29 14:55:29.444512 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.444244 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl"] Jul 29 14:55:29.444512 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.444480 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" podUID="2f4b781d-77be-4920-a063-30cfa173cc94" containerName="payload-pre-processing" containerID="cri-o://abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3" gracePeriod=30 Jul 29 14:55:29.733466 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.733446 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:55:29.736472 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.736451 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:55:29.891572 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.891536 2549 generic.go:356] "Generic (PLEG): container finished" podID="2f4b781d-77be-4920-a063-30cfa173cc94" containerID="abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3" exitCode=0 Jul 29 14:55:29.891762 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.891610 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" Jul 29 14:55:29.891762 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.891632 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" event={"ID":"2f4b781d-77be-4920-a063-30cfa173cc94","Type":"ContainerDied","Data":"abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3"} Jul 29 14:55:29.891762 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.891688 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl" event={"ID":"2f4b781d-77be-4920-a063-30cfa173cc94","Type":"ContainerDied","Data":"d097c4f5483228d901361648bb3d6b8fe739ade93b182c7520475518fea7aa78"} Jul 29 14:55:29.891762 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.891717 2549 scope.go:117] "RemoveContainer" containerID="abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3" Jul 29 14:55:29.892725 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.892692 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkqn4\" (UniqueName: \"kubernetes.io/projected/2f4b781d-77be-4920-a063-30cfa173cc94-kube-api-access-nkqn4\") pod \"2f4b781d-77be-4920-a063-30cfa173cc94\" (UID: \"2f4b781d-77be-4920-a063-30cfa173cc94\") " Jul 29 14:55:29.892793 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.892770 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkvws\" (UniqueName: \"kubernetes.io/projected/ac754512-9c9d-47ed-a15e-14f4450e8c00-kube-api-access-wkvws\") pod \"ac754512-9c9d-47ed-a15e-14f4450e8c00\" (UID: \"ac754512-9c9d-47ed-a15e-14f4450e8c00\") " Jul 29 14:55:29.892858 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.892805 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac754512-9c9d-47ed-a15e-14f4450e8c00-plugins-config-volume\") pod \"ac754512-9c9d-47ed-a15e-14f4450e8c00\" (UID: \"ac754512-9c9d-47ed-a15e-14f4450e8c00\") " Jul 29 14:55:29.892913 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.892889 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4b781d-77be-4920-a063-30cfa173cc94-plugins-config-volume\") pod \"2f4b781d-77be-4920-a063-30cfa173cc94\" (UID: \"2f4b781d-77be-4920-a063-30cfa173cc94\") " Jul 29 14:55:29.893245 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.893221 2549 generic.go:356] "Generic (PLEG): container finished" podID="ac754512-9c9d-47ed-a15e-14f4450e8c00" containerID="7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df" exitCode=0 Jul 29 14:55:29.893363 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.893248 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" event={"ID":"ac754512-9c9d-47ed-a15e-14f4450e8c00","Type":"ContainerDied","Data":"7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df"} Jul 29 14:55:29.893363 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.893292 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" Jul 29 14:55:29.893363 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.893332 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c" event={"ID":"ac754512-9c9d-47ed-a15e-14f4450e8c00","Type":"ContainerDied","Data":"7beb95633b9f9b6d294c38f21ed20775fcce35242e87c69072aba16f180356d7"} Jul 29 14:55:29.893536 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.893287 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac754512-9c9d-47ed-a15e-14f4450e8c00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac754512-9c9d-47ed-a15e-14f4450e8c00" (UID: "ac754512-9c9d-47ed-a15e-14f4450e8c00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:55:29.893895 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.893865 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4b781d-77be-4920-a063-30cfa173cc94-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f4b781d-77be-4920-a063-30cfa173cc94" (UID: "2f4b781d-77be-4920-a063-30cfa173cc94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:55:29.894897 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.894876 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac754512-9c9d-47ed-a15e-14f4450e8c00-kube-api-access-wkvws" (OuterVolumeSpecName: "kube-api-access-wkvws") pod "ac754512-9c9d-47ed-a15e-14f4450e8c00" (UID: "ac754512-9c9d-47ed-a15e-14f4450e8c00"). InnerVolumeSpecName "kube-api-access-wkvws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:55:29.894897 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.894884 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4b781d-77be-4920-a063-30cfa173cc94-kube-api-access-nkqn4" (OuterVolumeSpecName: "kube-api-access-nkqn4") pod "2f4b781d-77be-4920-a063-30cfa173cc94" (UID: "2f4b781d-77be-4920-a063-30cfa173cc94"). InnerVolumeSpecName "kube-api-access-nkqn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:55:29.907267 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.907250 2549 scope.go:117] "RemoveContainer" containerID="abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3" Jul 29 14:55:29.907533 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:55:29.907512 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3\": container with ID starting with abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3 not found: ID does not exist" containerID="abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3" Jul 29 14:55:29.907620 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.907540 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3"} err="failed to get container status \"abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3\": rpc error: code = NotFound desc = could not find container \"abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3\": container with ID starting with abb3f3a88f3627011db5a1dcd8da3a7c611c9d2bb7ece5a23cbe09991e3251c3 not found: ID does not exist" Jul 29 14:55:29.907620 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.907562 2549 scope.go:117] "RemoveContainer" containerID="7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df" Jul 29 14:55:29.914898 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.914883 2549 scope.go:117] "RemoveContainer" containerID="7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df" Jul 29 14:55:29.915108 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:55:29.915090 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df\": container with ID starting with 7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df not found: ID does not exist" containerID="7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df" Jul 29 14:55:29.915150 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.915114 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df"} err="failed to get container status \"7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df\": rpc error: code = NotFound desc = could not find container \"7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df\": container with ID starting with 7d6484ffb3581b6dd7749286ac4641c68ffad5141d6c3b00cdcd241cdf20d8df not found: ID does not exist" Jul 29 14:55:29.993781 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.993720 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4b781d-77be-4920-a063-30cfa173cc94-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:55:29.993781 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.993742 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkqn4\" (UniqueName: \"kubernetes.io/projected/2f4b781d-77be-4920-a063-30cfa173cc94-kube-api-access-nkqn4\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:55:29.993781 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.993756 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkvws\" (UniqueName: \"kubernetes.io/projected/ac754512-9c9d-47ed-a15e-14f4450e8c00-kube-api-access-wkvws\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:55:29.993781 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:29.993768 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac754512-9c9d-47ed-a15e-14f4450e8c00-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:55:30.221571 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:30.221545 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl"] Jul 29 14:55:30.227427 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:30.227404 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad0bed-85558cd676-wrcjl"] Jul 29 14:55:30.243151 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:30.243127 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c"] Jul 29 14:55:30.246345 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:30.246278 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad0bed-79c5bd794d-r4b7c"] Jul 29 14:55:30.866026 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:30.865992 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4b781d-77be-4920-a063-30cfa173cc94" path="/var/lib/kubelet/pods/2f4b781d-77be-4920-a063-30cfa173cc94/volumes" Jul 29 14:55:30.866407 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:30.866360 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac754512-9c9d-47ed-a15e-14f4450e8c00" path="/var/lib/kubelet/pods/ac754512-9c9d-47ed-a15e-14f4450e8c00/volumes" Jul 29 14:55:57.802376 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:57.802348 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr"] Jul 29 14:55:57.802924 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:57.802561 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" podUID="a5bd7ec1-eb80-43e5-9b78-546326d6656d" containerName="payload-processing" containerID="cri-o://2e6ed5b8d52c0ba4341212017234edc7981024644d97047baa271e980038c74b" gracePeriod=30 Jul 29 14:55:57.811642 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:57.811618 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd"] Jul 29 14:55:57.811814 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:57.811794 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" podUID="e88f6f4a-b78a-4a63-bb28-92c3d6635d76" containerName="payload-pre-processing" containerID="cri-o://30649a526be22382ec999e6da0d400aad308b2a2d2ba8aa56dc4b45c653e706c" gracePeriod=30 Jul 29 14:55:58.034627 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.034593 2549 generic.go:356] "Generic (PLEG): container finished" podID="e88f6f4a-b78a-4a63-bb28-92c3d6635d76" containerID="30649a526be22382ec999e6da0d400aad308b2a2d2ba8aa56dc4b45c653e706c" exitCode=0 Jul 29 14:55:58.035045 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.034818 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" event={"ID":"e88f6f4a-b78a-4a63-bb28-92c3d6635d76","Type":"ContainerDied","Data":"30649a526be22382ec999e6da0d400aad308b2a2d2ba8aa56dc4b45c653e706c"} Jul 29 14:55:58.037238 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.037199 2549 generic.go:356] "Generic (PLEG): container finished" podID="a5bd7ec1-eb80-43e5-9b78-546326d6656d" containerID="2e6ed5b8d52c0ba4341212017234edc7981024644d97047baa271e980038c74b" exitCode=0 Jul 29 14:55:58.037380 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.037357 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" event={"ID":"a5bd7ec1-eb80-43e5-9b78-546326d6656d","Type":"ContainerDied","Data":"2e6ed5b8d52c0ba4341212017234edc7981024644d97047baa271e980038c74b"} Jul 29 14:55:58.097465 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.097444 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:55:58.100649 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.100631 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:55:58.131148 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.131127 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg8r8\" (UniqueName: \"kubernetes.io/projected/a5bd7ec1-eb80-43e5-9b78-546326d6656d-kube-api-access-lg8r8\") pod \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\" (UID: \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\") " Jul 29 14:55:58.131236 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.131171 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-plugins-config-volume\") pod \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\" (UID: \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\") " Jul 29 14:55:58.131236 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.131207 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggm79\" (UniqueName: \"kubernetes.io/projected/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-kube-api-access-ggm79\") pod \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\" (UID: \"e88f6f4a-b78a-4a63-bb28-92c3d6635d76\") " Jul 29 14:55:58.131346 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.131289 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd7ec1-eb80-43e5-9b78-546326d6656d-plugins-config-volume\") pod \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\" (UID: \"a5bd7ec1-eb80-43e5-9b78-546326d6656d\") " Jul 29 14:55:58.131616 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.131578 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e88f6f4a-b78a-4a63-bb28-92c3d6635d76" (UID: "e88f6f4a-b78a-4a63-bb28-92c3d6635d76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:55:58.131706 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.131688 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bd7ec1-eb80-43e5-9b78-546326d6656d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5bd7ec1-eb80-43e5-9b78-546326d6656d" (UID: "a5bd7ec1-eb80-43e5-9b78-546326d6656d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:55:58.133207 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.133186 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-kube-api-access-ggm79" (OuterVolumeSpecName: "kube-api-access-ggm79") pod "e88f6f4a-b78a-4a63-bb28-92c3d6635d76" (UID: "e88f6f4a-b78a-4a63-bb28-92c3d6635d76"). InnerVolumeSpecName "kube-api-access-ggm79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:55:58.133264 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.133209 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bd7ec1-eb80-43e5-9b78-546326d6656d-kube-api-access-lg8r8" (OuterVolumeSpecName: "kube-api-access-lg8r8") pod "a5bd7ec1-eb80-43e5-9b78-546326d6656d" (UID: "a5bd7ec1-eb80-43e5-9b78-546326d6656d"). InnerVolumeSpecName "kube-api-access-lg8r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:55:58.232095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.232071 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:55:58.232095 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.232093 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggm79\" (UniqueName: \"kubernetes.io/projected/e88f6f4a-b78a-4a63-bb28-92c3d6635d76-kube-api-access-ggm79\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:55:58.232247 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.232106 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd7ec1-eb80-43e5-9b78-546326d6656d-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:55:58.232247 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:58.232114 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg8r8\" (UniqueName: \"kubernetes.io/projected/a5bd7ec1-eb80-43e5-9b78-546326d6656d-kube-api-access-lg8r8\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:55:59.043128 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.043096 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" Jul 29 14:55:59.043524 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.043131 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd" event={"ID":"e88f6f4a-b78a-4a63-bb28-92c3d6635d76","Type":"ContainerDied","Data":"1c4279d33194739507fd8e0117bd8d51681d0cf8a9595f089f950195b43dd365"} Jul 29 14:55:59.043524 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.043202 2549 scope.go:117] "RemoveContainer" containerID="30649a526be22382ec999e6da0d400aad308b2a2d2ba8aa56dc4b45c653e706c" Jul 29 14:55:59.044675 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.044651 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" event={"ID":"a5bd7ec1-eb80-43e5-9b78-546326d6656d","Type":"ContainerDied","Data":"efacb4ed44c360e095e6f978dcdbaa78c756204398c9207c9b7d99cbea9dd65d"} Jul 29 14:55:59.044766 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.044687 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr" Jul 29 14:55:59.052375 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.052358 2549 scope.go:117] "RemoveContainer" containerID="2e6ed5b8d52c0ba4341212017234edc7981024644d97047baa271e980038c74b" Jul 29 14:55:59.074655 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.074604 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr"] Jul 29 14:55:59.092050 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.092031 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19afc1-67bcfcc85b-d5rpr"] Jul 29 14:55:59.108671 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.108649 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd"] Jul 29 14:55:59.116802 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:55:59.116782 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19afc1-7d64b9b98d-2hxnd"] Jul 29 14:56:00.864786 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:00.864754 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5bd7ec1-eb80-43e5-9b78-546326d6656d" path="/var/lib/kubelet/pods/a5bd7ec1-eb80-43e5-9b78-546326d6656d/volumes" Jul 29 14:56:00.865260 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:00.865239 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e88f6f4a-b78a-4a63-bb28-92c3d6635d76" path="/var/lib/kubelet/pods/e88f6f4a-b78a-4a63-bb28-92c3d6635d76/volumes" Jul 29 14:56:15.046429 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:15.046334 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:56:15.052435 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:15.052411 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qnvb6_83f582eb-331b-4728-9e63-a5c94499d8b3/ovn-acl-logging/0.log" Jul 29 14:56:59.662300 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:59.662215 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x"] Jul 29 14:56:59.662838 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:59.662469 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" podUID="d4268033-0b47-40bb-b553-3a0d7a25c38b" containerName="payload-processing" containerID="cri-o://2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156" gracePeriod=30 Jul 29 14:56:59.666852 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:59.666823 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t"] Jul 29 14:56:59.667614 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:59.667560 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" podUID="35eaf4a4-ac5a-411e-b518-83bffd851a22" containerName="payload-pre-processing" containerID="cri-o://ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7" gracePeriod=30 Jul 29 14:56:59.957778 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:59.957750 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:56:59.962024 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:56:59.961997 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:57:00.041116 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.041077 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6hl9\" (UniqueName: \"kubernetes.io/projected/d4268033-0b47-40bb-b553-3a0d7a25c38b-kube-api-access-p6hl9\") pod \"d4268033-0b47-40bb-b553-3a0d7a25c38b\" (UID: \"d4268033-0b47-40bb-b553-3a0d7a25c38b\") " Jul 29 14:57:00.041301 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.041142 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eaf4a4-ac5a-411e-b518-83bffd851a22-plugins-config-volume\") pod \"35eaf4a4-ac5a-411e-b518-83bffd851a22\" (UID: \"35eaf4a4-ac5a-411e-b518-83bffd851a22\") " Jul 29 14:57:00.041301 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.041246 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4268033-0b47-40bb-b553-3a0d7a25c38b-plugins-config-volume\") pod \"d4268033-0b47-40bb-b553-3a0d7a25c38b\" (UID: \"d4268033-0b47-40bb-b553-3a0d7a25c38b\") " Jul 29 14:57:00.041440 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.041343 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj8gc\" (UniqueName: \"kubernetes.io/projected/35eaf4a4-ac5a-411e-b518-83bffd851a22-kube-api-access-nj8gc\") pod \"35eaf4a4-ac5a-411e-b518-83bffd851a22\" (UID: \"35eaf4a4-ac5a-411e-b518-83bffd851a22\") " Jul 29 14:57:00.041608 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.041579 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eaf4a4-ac5a-411e-b518-83bffd851a22-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35eaf4a4-ac5a-411e-b518-83bffd851a22" (UID: "35eaf4a4-ac5a-411e-b518-83bffd851a22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:00.041608 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.041597 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4268033-0b47-40bb-b553-3a0d7a25c38b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4268033-0b47-40bb-b553-3a0d7a25c38b" (UID: "d4268033-0b47-40bb-b553-3a0d7a25c38b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:00.041787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.041698 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35eaf4a4-ac5a-411e-b518-83bffd851a22-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:00.041787 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.041715 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4268033-0b47-40bb-b553-3a0d7a25c38b-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:00.043533 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.043491 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eaf4a4-ac5a-411e-b518-83bffd851a22-kube-api-access-nj8gc" (OuterVolumeSpecName: "kube-api-access-nj8gc") pod "35eaf4a4-ac5a-411e-b518-83bffd851a22" (UID: "35eaf4a4-ac5a-411e-b518-83bffd851a22"). InnerVolumeSpecName "kube-api-access-nj8gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:00.043629 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.043614 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4268033-0b47-40bb-b553-3a0d7a25c38b-kube-api-access-p6hl9" (OuterVolumeSpecName: "kube-api-access-p6hl9") pod "d4268033-0b47-40bb-b553-3a0d7a25c38b" (UID: "d4268033-0b47-40bb-b553-3a0d7a25c38b"). InnerVolumeSpecName "kube-api-access-p6hl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:00.142338 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.142285 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8gc\" (UniqueName: \"kubernetes.io/projected/35eaf4a4-ac5a-411e-b518-83bffd851a22-kube-api-access-nj8gc\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:00.142338 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.142335 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6hl9\" (UniqueName: \"kubernetes.io/projected/d4268033-0b47-40bb-b553-3a0d7a25c38b-kube-api-access-p6hl9\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:00.351447 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.351412 2549 generic.go:356] "Generic (PLEG): container finished" podID="35eaf4a4-ac5a-411e-b518-83bffd851a22" containerID="ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7" exitCode=0 Jul 29 14:57:00.351661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.351467 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" Jul 29 14:57:00.351661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.351508 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" event={"ID":"35eaf4a4-ac5a-411e-b518-83bffd851a22","Type":"ContainerDied","Data":"ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7"} Jul 29 14:57:00.351661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.351595 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t" event={"ID":"35eaf4a4-ac5a-411e-b518-83bffd851a22","Type":"ContainerDied","Data":"b78d1852363ca9b4d5ed28072f37c3889ca595c9c8b2cd0c17f2dea1321fea0b"} Jul 29 14:57:00.351661 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.351626 2549 scope.go:117] "RemoveContainer" containerID="ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7" Jul 29 14:57:00.352971 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.352944 2549 generic.go:356] "Generic (PLEG): container finished" podID="d4268033-0b47-40bb-b553-3a0d7a25c38b" containerID="2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156" exitCode=0 Jul 29 14:57:00.353082 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.352976 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" event={"ID":"d4268033-0b47-40bb-b553-3a0d7a25c38b","Type":"ContainerDied","Data":"2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156"} Jul 29 14:57:00.353082 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.353002 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" Jul 29 14:57:00.353082 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.353018 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x" event={"ID":"d4268033-0b47-40bb-b553-3a0d7a25c38b","Type":"ContainerDied","Data":"091f2bda6004a6fb545015b4e6f1e017e71ec5d8f7543b380a01ff04507283ef"} Jul 29 14:57:00.361249 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.361226 2549 scope.go:117] "RemoveContainer" containerID="ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7" Jul 29 14:57:00.361521 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:57:00.361500 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7\": container with ID starting with ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7 not found: ID does not exist" containerID="ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7" Jul 29 14:57:00.361612 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.361527 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7"} err="failed to get container status \"ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7\": rpc error: code = NotFound desc = could not find container \"ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7\": container with ID starting with ce6e82d9bcecf14a2a7edbc68a3c59184b32ac4fe82fbb4edf608027cc4e0bc7 not found: ID does not exist" Jul 29 14:57:00.361612 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.361544 2549 scope.go:117] "RemoveContainer" containerID="2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156" Jul 29 14:57:00.369505 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.369485 2549 scope.go:117] "RemoveContainer" containerID="2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156" Jul 29 14:57:00.369772 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:57:00.369753 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156\": container with ID starting with 2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156 not found: ID does not exist" containerID="2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156" Jul 29 14:57:00.369824 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.369781 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156"} err="failed to get container status \"2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156\": rpc error: code = NotFound desc = could not find container \"2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156\": container with ID starting with 2777be8e9e07d71784dda4a560304b94d71ec47c0c804ceafec8573a67ec5156 not found: ID does not exist" Jul 29 14:57:00.382382 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.382351 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x"] Jul 29 14:57:00.384145 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.384125 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-726aca-549778bd-7lk9x"] Jul 29 14:57:00.405378 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.405356 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t"] Jul 29 14:57:00.412514 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.412494 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-726aca-64d8c8d548-ph94t"] Jul 29 14:57:00.864709 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.864674 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35eaf4a4-ac5a-411e-b518-83bffd851a22" path="/var/lib/kubelet/pods/35eaf4a4-ac5a-411e-b518-83bffd851a22/volumes" Jul 29 14:57:00.865100 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:00.865034 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4268033-0b47-40bb-b553-3a0d7a25c38b" path="/var/lib/kubelet/pods/d4268033-0b47-40bb-b553-3a0d7a25c38b/volumes" Jul 29 14:57:17.925432 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:17.925398 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk"] Jul 29 14:57:17.925891 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:17.925648 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" podUID="303318c6-72ca-4a8e-b55f-8e46eabf875c" containerName="payload-processing" containerID="cri-o://75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb" gracePeriod=30 Jul 29 14:57:17.933165 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:17.933135 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8"] Jul 29 14:57:17.933397 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:17.933374 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" podUID="15f5659a-98b6-444e-a4eb-5ace33fd7b56" containerName="payload-pre-processing" containerID="cri-o://900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860" gracePeriod=30 Jul 29 14:57:17.953983 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:17.953942 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" podUID="303318c6-72ca-4a8e-b55f-8e46eabf875c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.54:9004: connect: connection refused" Jul 29 14:57:17.954108 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:17.953957 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" podUID="15f5659a-98b6-444e-a4eb-5ace33fd7b56" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.53:9004: connect: connection refused" Jul 29 14:57:18.213693 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.213673 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:57:18.216673 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.216653 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:57:18.296159 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.296137 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5whpx\" (UniqueName: \"kubernetes.io/projected/303318c6-72ca-4a8e-b55f-8e46eabf875c-kube-api-access-5whpx\") pod \"303318c6-72ca-4a8e-b55f-8e46eabf875c\" (UID: \"303318c6-72ca-4a8e-b55f-8e46eabf875c\") " Jul 29 14:57:18.296291 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.296193 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-422nh\" (UniqueName: \"kubernetes.io/projected/15f5659a-98b6-444e-a4eb-5ace33fd7b56-kube-api-access-422nh\") pod \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\" (UID: \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\") " Jul 29 14:57:18.296291 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.296236 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303318c6-72ca-4a8e-b55f-8e46eabf875c-plugins-config-volume\") pod \"303318c6-72ca-4a8e-b55f-8e46eabf875c\" (UID: \"303318c6-72ca-4a8e-b55f-8e46eabf875c\") " Jul 29 14:57:18.296291 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.296279 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5659a-98b6-444e-a4eb-5ace33fd7b56-plugins-config-volume\") pod \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\" (UID: \"15f5659a-98b6-444e-a4eb-5ace33fd7b56\") " Jul 29 14:57:18.296656 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.296632 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303318c6-72ca-4a8e-b55f-8e46eabf875c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "303318c6-72ca-4a8e-b55f-8e46eabf875c" (UID: "303318c6-72ca-4a8e-b55f-8e46eabf875c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:18.296762 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.296661 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f5659a-98b6-444e-a4eb-5ace33fd7b56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15f5659a-98b6-444e-a4eb-5ace33fd7b56" (UID: "15f5659a-98b6-444e-a4eb-5ace33fd7b56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:18.298182 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.298162 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303318c6-72ca-4a8e-b55f-8e46eabf875c-kube-api-access-5whpx" (OuterVolumeSpecName: "kube-api-access-5whpx") pod "303318c6-72ca-4a8e-b55f-8e46eabf875c" (UID: "303318c6-72ca-4a8e-b55f-8e46eabf875c"). InnerVolumeSpecName "kube-api-access-5whpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:18.298282 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.298256 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f5659a-98b6-444e-a4eb-5ace33fd7b56-kube-api-access-422nh" (OuterVolumeSpecName: "kube-api-access-422nh") pod "15f5659a-98b6-444e-a4eb-5ace33fd7b56" (UID: "15f5659a-98b6-444e-a4eb-5ace33fd7b56"). InnerVolumeSpecName "kube-api-access-422nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:18.397833 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.397811 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5whpx\" (UniqueName: \"kubernetes.io/projected/303318c6-72ca-4a8e-b55f-8e46eabf875c-kube-api-access-5whpx\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:18.397833 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.397832 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-422nh\" (UniqueName: \"kubernetes.io/projected/15f5659a-98b6-444e-a4eb-5ace33fd7b56-kube-api-access-422nh\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:18.397973 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.397844 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303318c6-72ca-4a8e-b55f-8e46eabf875c-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:18.397973 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.397853 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f5659a-98b6-444e-a4eb-5ace33fd7b56-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:18.453446 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.453385 2549 generic.go:356] "Generic (PLEG): container finished" podID="303318c6-72ca-4a8e-b55f-8e46eabf875c" containerID="75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb" exitCode=0 Jul 29 14:57:18.453446 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.453428 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" Jul 29 14:57:18.453602 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.453476 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" event={"ID":"303318c6-72ca-4a8e-b55f-8e46eabf875c","Type":"ContainerDied","Data":"75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb"} Jul 29 14:57:18.453602 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.453529 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk" event={"ID":"303318c6-72ca-4a8e-b55f-8e46eabf875c","Type":"ContainerDied","Data":"2922dea64eff2045c5b4aca681417babb54d04699768588ff5d6dfa762600b47"} Jul 29 14:57:18.453602 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.453555 2549 scope.go:117] "RemoveContainer" containerID="75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb" Jul 29 14:57:18.454845 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.454820 2549 generic.go:356] "Generic (PLEG): container finished" podID="15f5659a-98b6-444e-a4eb-5ace33fd7b56" containerID="900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860" exitCode=0 Jul 29 14:57:18.454938 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.454856 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" Jul 29 14:57:18.454995 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.454866 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" event={"ID":"15f5659a-98b6-444e-a4eb-5ace33fd7b56","Type":"ContainerDied","Data":"900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860"} Jul 29 14:57:18.455037 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.455013 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8" event={"ID":"15f5659a-98b6-444e-a4eb-5ace33fd7b56","Type":"ContainerDied","Data":"99be740d83afc1e0b33da0553f47157c7d1c63cf5ac61c2cfd70716e945b0f88"} Jul 29 14:57:18.462569 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.462537 2549 scope.go:117] "RemoveContainer" containerID="75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb" Jul 29 14:57:18.462902 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:57:18.462877 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb\": container with ID starting with 75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb not found: ID does not exist" containerID="75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb" Jul 29 14:57:18.462994 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.462914 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb"} err="failed to get container status \"75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb\": rpc error: code = NotFound desc = could not find container \"75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb\": container with ID starting with 75e0fde2b0471e50aa5556ab424bbc50128d048835b12c7dfbb9f614858235eb not found: ID does not exist" Jul 29 14:57:18.462994 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.462939 2549 scope.go:117] "RemoveContainer" containerID="900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860" Jul 29 14:57:18.471233 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.471211 2549 scope.go:117] "RemoveContainer" containerID="900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860" Jul 29 14:57:18.471519 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:57:18.471491 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860\": container with ID starting with 900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860 not found: ID does not exist" containerID="900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860" Jul 29 14:57:18.471591 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.471525 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860"} err="failed to get container status \"900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860\": rpc error: code = NotFound desc = could not find container \"900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860\": container with ID starting with 900e2c657d1e64ccb1feb106a14dc2c9ee8e4b919c26b2cff75646de47dd9860 not found: ID does not exist" Jul 29 14:57:18.479420 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.479399 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk"] Jul 29 14:57:18.482857 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.482836 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-61bc0a-5fc957c58-xbfhk"] Jul 29 14:57:18.493936 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.493915 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8"] Jul 29 14:57:18.497169 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.497150 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-61bc0a-599bdbcdcb-5lkg8"] Jul 29 14:57:18.865020 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.864991 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15f5659a-98b6-444e-a4eb-5ace33fd7b56" path="/var/lib/kubelet/pods/15f5659a-98b6-444e-a4eb-5ace33fd7b56/volumes" Jul 29 14:57:18.865374 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:18.865360 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="303318c6-72ca-4a8e-b55f-8e46eabf875c" path="/var/lib/kubelet/pods/303318c6-72ca-4a8e-b55f-8e46eabf875c/volumes" Jul 29 14:57:35.930399 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:35.930360 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn"] Jul 29 14:57:35.930996 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:35.930661 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" podUID="8135473d-43a2-4699-9729-66204f3a9138" containerName="payload-processing" containerID="cri-o://8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea" gracePeriod=30 Jul 29 14:57:35.936398 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:35.936297 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz"] Jul 29 14:57:35.936554 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:35.936525 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" podUID="a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8" containerName="payload-pre-processing" containerID="cri-o://9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4" gracePeriod=30 Jul 29 14:57:36.205893 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.205868 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:57:36.232730 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.232701 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:57:36.366983 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.366952 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-plugins-config-volume\") pod \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\" (UID: \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\") " Jul 29 14:57:36.367188 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.367012 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97w4w\" (UniqueName: \"kubernetes.io/projected/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-kube-api-access-97w4w\") pod \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\" (UID: \"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8\") " Jul 29 14:57:36.367188 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.367051 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbzcn\" (UniqueName: \"kubernetes.io/projected/8135473d-43a2-4699-9729-66204f3a9138-kube-api-access-tbzcn\") pod \"8135473d-43a2-4699-9729-66204f3a9138\" (UID: \"8135473d-43a2-4699-9729-66204f3a9138\") " Jul 29 14:57:36.367188 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.367085 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8135473d-43a2-4699-9729-66204f3a9138-plugins-config-volume\") pod \"8135473d-43a2-4699-9729-66204f3a9138\" (UID: \"8135473d-43a2-4699-9729-66204f3a9138\") " Jul 29 14:57:36.367513 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.367477 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8" (UID: "a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:36.367583 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.367541 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8135473d-43a2-4699-9729-66204f3a9138-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8135473d-43a2-4699-9729-66204f3a9138" (UID: "8135473d-43a2-4699-9729-66204f3a9138"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:36.369271 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.369247 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-kube-api-access-97w4w" (OuterVolumeSpecName: "kube-api-access-97w4w") pod "a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8" (UID: "a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8"). InnerVolumeSpecName "kube-api-access-97w4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:36.369401 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.369284 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8135473d-43a2-4699-9729-66204f3a9138-kube-api-access-tbzcn" (OuterVolumeSpecName: "kube-api-access-tbzcn") pod "8135473d-43a2-4699-9729-66204f3a9138" (UID: "8135473d-43a2-4699-9729-66204f3a9138"). InnerVolumeSpecName "kube-api-access-tbzcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:36.468165 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.468082 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:36.468165 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.468108 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97w4w\" (UniqueName: \"kubernetes.io/projected/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8-kube-api-access-97w4w\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:36.468165 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.468117 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbzcn\" (UniqueName: \"kubernetes.io/projected/8135473d-43a2-4699-9729-66204f3a9138-kube-api-access-tbzcn\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:36.468165 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.468127 2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8135473d-43a2-4699-9729-66204f3a9138-plugins-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Jul 29 14:57:36.543285 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.543253 2549 generic.go:356] "Generic (PLEG): container finished" podID="a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8" containerID="9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4" exitCode=0 Jul 29 14:57:36.543285 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.543291 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" Jul 29 14:57:36.543545 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.543342 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" event={"ID":"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8","Type":"ContainerDied","Data":"9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4"} Jul 29 14:57:36.543545 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.543404 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz" event={"ID":"a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8","Type":"ContainerDied","Data":"3db33c36954de953703c9bf3e4d3e7534518235aab4514276bdf5e3b77f90d40"} Jul 29 14:57:36.543545 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.543436 2549 scope.go:117] "RemoveContainer" containerID="9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4" Jul 29 14:57:36.544690 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.544666 2549 generic.go:356] "Generic (PLEG): container finished" podID="8135473d-43a2-4699-9729-66204f3a9138" containerID="8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea" exitCode=0 Jul 29 14:57:36.544767 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.544722 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" event={"ID":"8135473d-43a2-4699-9729-66204f3a9138","Type":"ContainerDied","Data":"8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea"} Jul 29 14:57:36.544767 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.544727 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" Jul 29 14:57:36.544767 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.544756 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn" event={"ID":"8135473d-43a2-4699-9729-66204f3a9138","Type":"ContainerDied","Data":"b309efadafee4f0f87209ec6355b7a02404a5fe093c2e05919373f21ba47a5af"} Jul 29 14:57:36.552008 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.551987 2549 scope.go:117] "RemoveContainer" containerID="9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4" Jul 29 14:57:36.552286 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:57:36.552268 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4\": container with ID starting with 9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4 not found: ID does not exist" containerID="9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4" Jul 29 14:57:36.552373 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.552294 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4"} err="failed to get container status \"9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4\": rpc error: code = NotFound desc = could not find container \"9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4\": container with ID starting with 9dd91e562daaabc7002b401af8aa96d6f21cca5015dcfbbc33f1a16d848a93f4 not found: ID does not exist" Jul 29 14:57:36.552373 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.552334 2549 scope.go:117] "RemoveContainer" containerID="8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea" Jul 29 14:57:36.560796 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.560775 2549 scope.go:117] "RemoveContainer" containerID="8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea" Jul 29 14:57:36.561053 ip-10-0-130-28 kubenswrapper[2549]: E0729 14:57:36.561034 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea\": container with ID starting with 8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea not found: ID does not exist" containerID="8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea" Jul 29 14:57:36.561118 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.561059 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea"} err="failed to get container status \"8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea\": rpc error: code = NotFound desc = could not find container \"8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea\": container with ID starting with 8c50a85054fbaf21e5e6bec632d472479f2b759d9fa3025d61077e9b259f6cea not found: ID does not exist" Jul 29 14:57:36.570704 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.570683 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz"] Jul 29 14:57:36.576950 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.576929 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-532c20-dbf744dbd-rdqbz"] Jul 29 14:57:36.614986 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.614956 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn"] Jul 29 14:57:36.616507 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.616487 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-532c20-7ccfb78df4-77qhn"] Jul 29 14:57:36.864588 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.864555 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8135473d-43a2-4699-9729-66204f3a9138" path="/var/lib/kubelet/pods/8135473d-43a2-4699-9729-66204f3a9138/volumes" Jul 29 14:57:36.864920 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:36.864907 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8" path="/var/lib/kubelet/pods/a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8/volumes" Jul 29 14:57:49.273754 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:49.273723 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d87754bdc-lxjx5_1664efef-af3b-432c-a26c-5c16db455cb8/manager/0.log" Jul 29 14:57:49.925981 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:49.925949 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m_5fc5bb27-b105-441d-b3eb-33a1eea756e8/copy-manifests/0.log" Jul 29 14:57:49.938850 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:49.938824 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m_5fc5bb27-b105-441d-b3eb-33a1eea756e8/manager/0.log" Jul 29 14:57:52.990229 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:52.990195 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-qbhjn_57e4ef31-edff-46ff-aebe-944de3c05fe9/manager/0.log" Jul 29 14:57:53.462371 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:53.462334 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-79nlp_ce73abf6-8088-4d56-9338-57d2095a1ad6/discovery/0.log" Jul 29 14:57:53.926656 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:53.926620 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tmpvk_76c7b8ab-30aa-4387-bc0a-3b267b46100a/payload-pre-processing/0.log" Jul 29 14:57:54.046593 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:54.046560 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qswsb_0b59c1bd-570e-455f-89d7-d9e7a2856d63/payload-processing/0.log" Jul 29 14:57:54.737166 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:54.737133 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9gsn5_6ff07e17-7f6c-4ddf-b41d-7f736a663d71/main/0.log" Jul 29 14:57:54.848692 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:57:54.848659 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb8bf4_c074b6d8-919f-41f6-b62b-ddc5fd80134b/main/0.log" Jul 29 14:58:03.614187 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:03.614154 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dwzkb_ba453020-a7c0-4204-988e-cfc912eafadb/global-pull-secret-syncer/0.log" Jul 29 14:58:03.710477 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:03.710443 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8dml7_06d8cf0b-e7fd-4c62-a798-c6e26bda4526/konnectivity-agent/0.log" Jul 29 14:58:03.782550 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:03.782520 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-28.ec2.internal_aa70b4877e9abd3c146a0f6d52419063/haproxy/0.log" Jul 29 14:58:10.411406 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:10.411373 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-qbhjn_57e4ef31-edff-46ff-aebe-944de3c05fe9/manager/0.log" Jul 29 14:58:12.400975 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.400941 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-p82s4_010ceb81-e2a2-4531-94c2-1f8b03bdabec/kube-state-metrics/0.log" Jul 29 14:58:12.426045 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.426013 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-p82s4_010ceb81-e2a2-4531-94c2-1f8b03bdabec/kube-rbac-proxy-main/0.log" Jul 29 14:58:12.449947 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.449917 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-p82s4_010ceb81-e2a2-4531-94c2-1f8b03bdabec/kube-rbac-proxy-self/0.log" Jul 29 14:58:12.479878 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.479849 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-848c946485-6r9bg_a5b43b8f-e98f-4c6a-9e9c-f83145239ba5/metrics-server/0.log" Jul 29 14:58:12.723017 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.722929 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltcpg_992ee83e-02f2-4a61-b574-89b3e33177ef/node-exporter/0.log" Jul 29 14:58:12.746579 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.746548 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltcpg_992ee83e-02f2-4a61-b574-89b3e33177ef/kube-rbac-proxy/0.log" Jul 29 14:58:12.770039 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.770008 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltcpg_992ee83e-02f2-4a61-b574-89b3e33177ef/init-textfile/0.log" Jul 29 14:58:12.796012 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.795982 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nx4x4_a0827515-4442-4ff1-b78a-03647cf0afcc/kube-rbac-proxy-main/0.log" Jul 29 14:58:12.816701 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.816674 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nx4x4_a0827515-4442-4ff1-b78a-03647cf0afcc/kube-rbac-proxy-self/0.log" Jul 29 14:58:12.838729 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.838700 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nx4x4_a0827515-4442-4ff1-b78a-03647cf0afcc/openshift-state-metrics/0.log" Jul 29 14:58:12.890146 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.890113 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_67cfc846-94af-45ca-92ac-0879dd97b970/prometheus/0.log" Jul 29 14:58:12.913790 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.913763 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_67cfc846-94af-45ca-92ac-0879dd97b970/config-reloader/0.log" Jul 29 14:58:12.938489 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.938458 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_67cfc846-94af-45ca-92ac-0879dd97b970/thanos-sidecar/0.log" Jul 29 14:58:12.962024 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.962000 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_67cfc846-94af-45ca-92ac-0879dd97b970/kube-rbac-proxy-web/0.log" Jul 29 14:58:12.986508 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:12.986391 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_67cfc846-94af-45ca-92ac-0879dd97b970/kube-rbac-proxy/0.log" Jul 29 14:58:13.010161 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.010130 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_67cfc846-94af-45ca-92ac-0879dd97b970/kube-rbac-proxy-thanos/0.log" Jul 29 14:58:13.032961 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.032924 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_67cfc846-94af-45ca-92ac-0879dd97b970/init-config-reloader/0.log" Jul 29 14:58:13.112574 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.112530 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-drvnx_42bedec4-cbdd-49ff-934a-e6d0144d18ab/prometheus-operator-admission-webhook/0.log" Jul 29 14:58:13.142179 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.142154 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f88b49d44-hjs9z_634e491a-10ea-4c8a-b1d2-a1b41b910e90/telemeter-client/0.log" Jul 29 14:58:13.187917 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.187890 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f88b49d44-hjs9z_634e491a-10ea-4c8a-b1d2-a1b41b910e90/reload/0.log" Jul 29 14:58:13.210299 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.210267 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f88b49d44-hjs9z_634e491a-10ea-4c8a-b1d2-a1b41b910e90/kube-rbac-proxy/0.log" Jul 29 14:58:13.973222 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973184 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp"] Jul 29 14:58:13.973622 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973576 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88f6f4a-b78a-4a63-bb28-92c3d6635d76" containerName="payload-pre-processing" Jul 29 14:58:13.973622 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973588 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88f6f4a-b78a-4a63-bb28-92c3d6635d76" containerName="payload-pre-processing" Jul 29 14:58:13.973622 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973601 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8" containerName="payload-pre-processing" Jul 29 14:58:13.973622 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973607 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8" containerName="payload-pre-processing" Jul 29 14:58:13.973622 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973616 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303318c6-72ca-4a8e-b55f-8e46eabf875c" containerName="payload-processing" Jul 29 14:58:13.973622 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973621 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="303318c6-72ca-4a8e-b55f-8e46eabf875c" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973629 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35eaf4a4-ac5a-411e-b518-83bffd851a22" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973634 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="35eaf4a4-ac5a-411e-b518-83bffd851a22" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973645 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4268033-0b47-40bb-b553-3a0d7a25c38b" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973651 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4268033-0b47-40bb-b553-3a0d7a25c38b" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973657 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac754512-9c9d-47ed-a15e-14f4450e8c00" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973663 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac754512-9c9d-47ed-a15e-14f4450e8c00" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973670 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f5659a-98b6-444e-a4eb-5ace33fd7b56" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973675 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f5659a-98b6-444e-a4eb-5ace33fd7b56" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973681 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4b781d-77be-4920-a063-30cfa173cc94" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973686 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4b781d-77be-4920-a063-30cfa173cc94" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973693 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8135473d-43a2-4699-9729-66204f3a9138" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973699 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8135473d-43a2-4699-9729-66204f3a9138" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973708 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5bd7ec1-eb80-43e5-9b78-546326d6656d" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973713 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5bd7ec1-eb80-43e5-9b78-546326d6656d" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973768 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8135473d-43a2-4699-9729-66204f3a9138" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973777 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4268033-0b47-40bb-b553-3a0d7a25c38b" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973783 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f5659a-98b6-444e-a4eb-5ace33fd7b56" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973788 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5bd7ec1-eb80-43e5-9b78-546326d6656d" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973796 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d093f2-69f9-4eed-a1b5-9a0da8f30fd8" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973801 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88f6f4a-b78a-4a63-bb28-92c3d6635d76" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973808 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="303318c6-72ca-4a8e-b55f-8e46eabf875c" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973814 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac754512-9c9d-47ed-a15e-14f4450e8c00" containerName="payload-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973820 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="35eaf4a4-ac5a-411e-b518-83bffd851a22" containerName="payload-pre-processing" Jul 29 14:58:13.973964 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.973825 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4b781d-77be-4920-a063-30cfa173cc94" containerName="payload-pre-processing" Jul 29 14:58:13.977123 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.977106 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:13.979817 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.979797 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpxzz\"/\"openshift-service-ca.crt\"" Jul 29 14:58:13.980808 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.980793 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpxzz\"/\"kube-root-ca.crt\"" Jul 29 14:58:13.980885 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.980795 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tpxzz\"/\"default-dockercfg-2v4xz\"" Jul 29 14:58:13.986666 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:13.986641 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp"] Jul 29 14:58:14.098180 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.098135 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-sys\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.098418 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.098188 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9xw\" (UniqueName: \"kubernetes.io/projected/202e4ef5-84e6-4a87-8a6a-4f572a48355c-kube-api-access-4n9xw\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.098418 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.098291 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-proc\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.098553 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.098418 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-podres\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.098553 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.098469 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-lib-modules\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.199712 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.199667 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9xw\" (UniqueName: \"kubernetes.io/projected/202e4ef5-84e6-4a87-8a6a-4f572a48355c-kube-api-access-4n9xw\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.199902 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.199738 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-proc\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.199902 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.199779 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-podres\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.199902 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.199807 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-lib-modules\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.199902 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.199835 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-sys\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.199902 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.199881 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-proc\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.200162 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.199913 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-sys\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.200162 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.199944 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-podres\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.200162 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.199962 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/202e4ef5-84e6-4a87-8a6a-4f572a48355c-lib-modules\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.208860 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.208826 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9xw\" (UniqueName: \"kubernetes.io/projected/202e4ef5-84e6-4a87-8a6a-4f572a48355c-kube-api-access-4n9xw\") pod \"perf-node-gather-daemonset-gttfp\" (UID: \"202e4ef5-84e6-4a87-8a6a-4f572a48355c\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.287725 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.287688 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.427790 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.427761 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp"] Jul 29 14:58:14.728285 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.728245 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" event={"ID":"202e4ef5-84e6-4a87-8a6a-4f572a48355c","Type":"ContainerStarted","Data":"1bcc5858737d149e0884a72dc1c78d92275655e4cf4e152aced946dabe1faf5f"} Jul 29 14:58:14.728460 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.728415 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:14.728460 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.728451 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" event={"ID":"202e4ef5-84e6-4a87-8a6a-4f572a48355c","Type":"ContainerStarted","Data":"0cd4ccbd7ce586761164b9b1fb8b1af8b1fd98f080bca790f209ee3ae1727a04"} Jul 29 14:58:14.746111 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:14.746040 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" podStartSLOduration=1.746024726 podStartE2EDuration="1.746024726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:58:14.744474461 +0000 UTC m=+3420.397068296" watchObservedRunningTime="2026-07-29 14:58:14.746024726 +0000 UTC m=+3420.398618560" Jul 29 14:58:15.677263 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:15.677181 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-5wwjd_83ad3cff-e751-4953-b740-70df9ce69f39/download-server/0.log" Jul 29 14:58:17.303064 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:17.303034 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kvbf4_afb32e32-1d93-4bf0-8134-7df7b7e3bfb6/dns/0.log" Jul 29 14:58:17.348139 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:17.348115 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kvbf4_afb32e32-1d93-4bf0-8134-7df7b7e3bfb6/kube-rbac-proxy/0.log" Jul 29 14:58:17.531237 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:17.531207 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdzkw_7e3ccd9f-be11-45b0-8380-fe38fdd54579/dns-node-resolver/0.log" Jul 29 14:58:18.082884 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:18.082856 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c7f585cd7-xf89c_d17adb8d-ff91-4c2d-aa69-5e7bb71f2eb9/registry/0.log" Jul 29 14:58:18.152504 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:18.152470 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vfwlj_4d8ecdc2-2c1a-47a7-b9a0-97439a6d3436/node-ca/0.log" Jul 29 14:58:19.302333 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:19.302294 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-79nlp_ce73abf6-8088-4d56-9338-57d2095a1ad6/discovery/0.log" Jul 29 14:58:19.475027 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:19.474998 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tmpvk_76c7b8ab-30aa-4387-bc0a-3b267b46100a/payload-pre-processing/0.log" Jul 29 14:58:19.512867 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:19.512837 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qswsb_0b59c1bd-570e-455f-89d7-d9e7a2856d63/payload-processing/0.log" Jul 29 14:58:20.354554 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:20.354522 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6499h_5f409ba5-0292-42bd-8997-7437b42b5a36/serve-healthcheck-canary/0.log" Jul 29 14:58:20.743044 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:20.742961 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-gttfp" Jul 29 14:58:20.918988 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:20.918953 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hgjg_5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef/kube-rbac-proxy/0.log" Jul 29 14:58:20.947864 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:20.947841 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hgjg_5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef/exporter/0.log" Jul 29 14:58:20.974448 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:20.974420 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hgjg_5c056bdd-bf6c-43b2-b5e7-63a8fa3d2fef/extractor/0.log" Jul 29 14:58:24.100351 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:24.100299 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d87754bdc-lxjx5_1664efef-af3b-432c-a26c-5c16db455cb8/manager/0.log" Jul 29 14:58:24.509192 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:24.509164 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m_5fc5bb27-b105-441d-b3eb-33a1eea756e8/manager/0.log" Jul 29 14:58:24.533573 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:24.533546 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff7cbc5f5-wnq4m_5fc5bb27-b105-441d-b3eb-33a1eea756e8/copy-manifests/0.log" Jul 29 14:58:25.261239 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:25.261206 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-79nlp_ce73abf6-8088-4d56-9338-57d2095a1ad6/discovery/0.log" Jul 29 14:58:25.399208 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:25.399123 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tmpvk_76c7b8ab-30aa-4387-bc0a-3b267b46100a/payload-pre-processing/0.log" Jul 29 14:58:25.421721 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:25.421691 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qswsb_0b59c1bd-570e-455f-89d7-d9e7a2856d63/payload-processing/0.log" Jul 29 14:58:25.903445 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:25.903415 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f9f5fc96c-f7qsf_a0866617-d190-4c22-85eb-cda2a4b0ec0e/manager/0.log" Jul 29 14:58:26.968123 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:26.968091 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-79nlp_ce73abf6-8088-4d56-9338-57d2095a1ad6/discovery/0.log" Jul 29 14:58:27.114489 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:27.114453 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tmpvk_76c7b8ab-30aa-4387-bc0a-3b267b46100a/payload-pre-processing/0.log" Jul 29 14:58:27.149280 ip-10-0-130-28 kubenswrapper[2549]: I0729 14:58:27.149243 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qswsb_0b59c1bd-570e-455f-89d7-d9e7a2856d63/payload-processing/0.log"