Jul 07 19:08:13.949196 ip-10-0-132-59 systemd[1]: Starting Kubernetes Kubelet... Jul 07 19:08:14.328678 ip-10-0-132-59 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:08:14.328678 ip-10-0-132-59 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 19:08:14.328678 ip-10-0-132-59 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:08:14.328678 ip-10-0-132-59 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 19:08:14.328678 ip-10-0-132-59 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:08:14.330114 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.330029 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 19:08:14.335337 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335322 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:08:14.335337 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335337 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335341 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335344 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335347 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335350 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335352 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335360 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335363 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335365 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335368 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335370 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335373 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335375 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335378 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335380 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335383 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335385 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335388 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335390 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335393 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:08:14.335398 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335396 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335398 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335400 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335403 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335406 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335409 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335411 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335416 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335419 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335423 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335425 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335428 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335431 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335434 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335436 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335438 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335441 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335443 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335446 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:08:14.335868 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335449 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335452 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335454 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335457 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335459 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335461 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335464 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335466 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335468 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335471 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335473 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335476 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335479 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335481 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335484 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335487 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335490 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335494 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335498 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335502 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:08:14.336371 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335504 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335507 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335509 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335512 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335514 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335516 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335519 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335521 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335524 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335526 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335529 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335531 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335533 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335536 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335539 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335541 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335543 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335546 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335548 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335551 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:08:14.336851 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335553 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335555 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335558 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335562 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335566 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335568 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335571 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335575 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335578 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335580 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335583 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335586 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335588 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335591 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335594 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335597 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335600 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335603 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.335605 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:08:14.337340 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336297 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336303 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336306 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336308 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336312 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336315 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336317 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336327 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336330 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336333 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336335 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336338 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336340 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336343 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336346 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336348 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336351 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336354 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336356 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336359 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:08:14.337796 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336361 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336364 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336367 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336369 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336372 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336375 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336377 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336380 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336382 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336385 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336388 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336390 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336392 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336395 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336397 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336399 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336402 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336404 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336406 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336409 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:08:14.338298 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336411 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336414 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336416 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336419 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336421 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336423 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336426 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336428 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336431 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336433 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336436 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336439 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336441 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336447 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336451 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336454 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336456 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336459 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336462 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:08:14.338781 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336464 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336468 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336471 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336474 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336476 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336479 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336481 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336483 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336486 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336488 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336491 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336493 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336495 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336498 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336500 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336503 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336505 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336507 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336509 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:08:14.339280 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336512 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336515 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336521 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336523 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336526 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336529 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336531 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336533 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336535 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336538 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336540 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336542 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336546 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336550 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336553 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336555 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336558 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336561 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336563 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:08:14.339728 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336566 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336568 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336932 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336937 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336939 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336942 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336945 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336947 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336950 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336952 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336955 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336958 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336960 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336963 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336965 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336969 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336972 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336975 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336977 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336980 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:08:14.340223 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336983 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336985 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336988 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336990 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336995 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.336998 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337001 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337004 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337007 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337010 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337012 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337015 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337017 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337020 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337022 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337024 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337027 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337029 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337032 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337034 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:08:14.340695 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337037 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337040 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337042 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337044 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337047 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337049 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337053 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337058 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337061 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337064 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337068 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337070 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337073 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337076 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337078 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337080 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337083 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337086 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337088 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:08:14.341200 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337091 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337093 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337095 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337098 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337100 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337102 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337105 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337107 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337109 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337112 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337115 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337117 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337119 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337122 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337125 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337128 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337131 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337133 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337135 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337138 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:08:14.341691 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337142 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337144 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337148 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337151 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337153 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337155 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337158 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337160 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337163 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337165 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337167 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337170 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337172 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337175 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337177 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337180 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337182 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337185 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337187 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337189 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:08:14.342198 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337192 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337194 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337272 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337282 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337293 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337300 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337304 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337307 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337312 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337316 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337319 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337321 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337326 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337330 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337334 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337337 2577 flags.go:64] FLAG: --cgroup-root="" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337340 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337343 2577 flags.go:64] FLAG: --client-ca-file="" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337346 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337348 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337352 2577 flags.go:64] FLAG: --cluster-domain="" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337355 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337358 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337361 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 19:08:14.342738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337364 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337368 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337370 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337373 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337376 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337379 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337382 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337384 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337387 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337390 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337394 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337396 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337399 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337402 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337404 2577 flags.go:64] FLAG: --enable-server="true" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337407 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337439 2577 flags.go:64] FLAG: --event-burst="100" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337443 2577 flags.go:64] FLAG: --event-qps="50" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337446 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337449 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337453 2577 flags.go:64] FLAG: --eviction-hard="" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337457 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337461 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337463 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337466 2577 flags.go:64] FLAG: --eviction-soft="" Jul 07 19:08:14.343485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337469 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337472 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337475 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337477 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337480 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337482 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337485 2577 flags.go:64] FLAG: --feature-gates="" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337489 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337492 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337494 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337497 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337500 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337503 2577 flags.go:64] FLAG: --help="false" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337506 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337509 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337512 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337514 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337517 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337520 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337523 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337525 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337528 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337530 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 19:08:14.344106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337533 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337536 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337539 2577 flags.go:64] FLAG: --kube-reserved="" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337542 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337546 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337548 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337552 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337555 2577 flags.go:64] FLAG: --lock-file="" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337558 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337560 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337563 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337569 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337572 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337574 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337577 2577 flags.go:64] FLAG: --logging-format="text" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337580 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337583 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337585 2577 flags.go:64] FLAG: --manifest-url="" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337588 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337592 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337595 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337598 2577 flags.go:64] FLAG: --max-pods="110" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337601 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337604 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337606 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 19:08:14.344685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337609 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337611 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337614 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337617 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337623 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337626 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337629 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337631 2577 flags.go:64] FLAG: --pod-cidr="" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337634 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337639 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337643 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337646 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337649 2577 flags.go:64] FLAG: --port="10250" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337654 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337657 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08f77fc6f1aa607e2" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337660 2577 flags.go:64] FLAG: --qos-reserved="" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337663 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337666 2577 flags.go:64] FLAG: --register-node="true" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337669 2577 flags.go:64] FLAG: --register-with-taints="" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337672 2577 flags.go:64] FLAG: --registry-burst="10" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337675 2577 flags.go:64] FLAG: --registry-qps="5" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337678 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337680 2577 flags.go:64] FLAG: --reserved-memory="" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337684 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337687 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 19:08:14.345312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337689 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337692 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337695 2577 flags.go:64] FLAG: --runonce="false" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337697 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337700 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337703 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337706 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337708 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337711 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337714 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337717 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337719 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337722 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337724 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337727 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337730 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337733 2577 flags.go:64] FLAG: --system-cgroups="" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337735 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337740 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337745 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337748 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337752 2577 flags.go:64] FLAG: --tls-min-version="" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337755 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337757 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337760 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 19:08:14.346023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337763 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337765 2577 flags.go:64] FLAG: --v="2" Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337769 2577 flags.go:64] FLAG: --version="false" Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337773 2577 flags.go:64] FLAG: --vmodule="" Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337776 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.337779 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337911 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337916 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337918 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337921 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337923 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337926 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337928 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337931 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337933 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337935 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337939 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337942 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337945 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337947 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337949 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:08:14.346945 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337952 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337954 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337957 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337960 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337962 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337968 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337972 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337974 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337977 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337979 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337982 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337984 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337987 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337989 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337991 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337994 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337996 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.337999 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338001 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338003 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:08:14.347486 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338005 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338008 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338010 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338013 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338015 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338018 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338020 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338022 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338025 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338027 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338030 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338032 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338034 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338037 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338039 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338041 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338044 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338046 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338050 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338052 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338055 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:08:14.347996 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338057 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338060 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338062 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338064 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338067 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338069 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338072 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338074 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338077 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338079 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338081 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338084 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338086 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338088 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338091 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338093 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338095 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338098 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338100 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338102 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:08:14.348516 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338105 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338107 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338109 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338113 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338116 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338119 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338121 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338124 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338127 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338131 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338135 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338138 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338141 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338143 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338146 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338148 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338151 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338153 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338155 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:08:14.349013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338158 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338160 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338163 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.338165 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.338824 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.347387 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.347400 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.347952 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.347967 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348026 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348031 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348035 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348040 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348044 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:08:14.349472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348048 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348052 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348055 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348058 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348061 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348063 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348066 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348069 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348072 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348074 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348077 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348079 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348081 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348084 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348088 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348091 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348094 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348097 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348100 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:08:14.349810 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348103 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348106 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348109 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348113 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348119 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348124 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348127 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348129 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348132 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348135 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348137 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348140 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348142 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348144 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348147 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348150 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348152 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348156 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348159 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:08:14.350292 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348162 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348164 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348167 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348170 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348173 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348175 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348178 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348180 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348183 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348185 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348188 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348193 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348197 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348201 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348204 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348207 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348209 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348212 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348215 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348218 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:08:14.350739 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348220 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348223 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348225 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348228 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348230 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348232 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348235 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348237 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348239 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348241 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348244 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348246 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348248 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348251 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348253 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348255 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348259 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348261 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348263 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348267 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:08:14.351323 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348271 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348276 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348280 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348283 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348285 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348288 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348290 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348293 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348295 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348298 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348300 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348303 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348305 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348308 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348310 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:08:14.351789 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348313 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.348317 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348443 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348449 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348452 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348455 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348458 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348460 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348463 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348465 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348468 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348471 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348473 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348476 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348479 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:08:14.352163 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348481 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348484 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348487 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348490 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348492 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348495 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348497 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348501 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348505 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348510 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348514 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348517 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348519 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348522 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348525 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348527 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348530 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348532 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348534 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348537 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:08:14.352543 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348539 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348541 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348544 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348546 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348549 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348551 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348555 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348559 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348562 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348564 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348567 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348570 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348574 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348577 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348581 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348586 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348590 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348593 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348595 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:08:14.353029 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348598 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348600 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348603 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348605 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348607 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348610 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348612 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348615 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348617 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348620 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348622 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348624 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348627 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348629 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348632 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348634 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348637 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348639 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348641 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348644 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:08:14.353472 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348646 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348649 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348653 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348657 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348661 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348665 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348670 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348673 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348675 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348678 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348680 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348683 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348685 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348689 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348692 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348694 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348697 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348699 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348702 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:08:14.353968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348704 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348707 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348709 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348712 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348714 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348717 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348719 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:14.348721 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.348726 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.348834 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.351082 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.352624 2577 server.go:1019] "Starting client certificate rotation" Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.352720 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:08:14.354443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.352751 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:08:14.355003 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.354990 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:08:14.357690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.357674 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:08:14.370487 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.370468 2577 log.go:25] "Validated CRI v1 runtime API" Jul 07 19:08:14.372669 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.372654 2577 log.go:25] "Validated CRI v1 image API" Jul 07 19:08:14.373418 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.373406 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 19:08:14.375757 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.375741 2577 fs.go:135] Filesystem UUIDs: map[1d03b7b3-8a27-4adf-86f6-78c524149a5e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b289e07b-bfe8-4015-b818-be12c80f284f:/dev/nvme0n1p3] Jul 07 19:08:14.375806 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.375758 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 19:08:14.381280 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.381117 2577 manager.go:216] Machine: {Timestamp:2026-07-07 19:08:14.379955038 +0000 UTC m=+0.315580389 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099351 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec202dfe94b940317f0ec70485a9d0ab SystemUUID:ec202dfe-94b9-4031-7f0e-c70485a9d0ab BootID:c0220373-8c39-4680-a567-2890f3644cda Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dd:94:56:08:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dd:94:56:08:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:d2:89:57:03:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 19:08:14.381280 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.381272 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 19:08:14.381383 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.381354 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 19:08:14.383063 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.383036 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 19:08:14.383219 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.383067 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-59.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 19:08:14.383920 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.383908 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 19:08:14.383972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.383931 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 19:08:14.383972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.383947 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:08:14.384085 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.384068 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:08:14.384513 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.384500 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:08:14.386375 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.386363 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 19:08:14.389860 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.389846 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:08:14.390086 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.390073 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 19:08:14.390951 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.390930 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 07 19:08:14.391020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.390955 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 19:08:14.391020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.390977 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 19:08:14.391020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.390990 2577 kubelet.go:398] "Adding apiserver pod source" Jul 07 19:08:14.391020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.391006 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 19:08:14.391491 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.391478 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:08:14.391543 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.391504 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:08:14.394319 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.394301 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 19:08:14.396690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.396674 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 19:08:14.396764 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.396704 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 19:08:14.397894 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397868 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 19:08:14.397940 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397899 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 19:08:14.397940 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397909 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 19:08:14.397940 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397917 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 19:08:14.397940 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397927 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 19:08:14.397940 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397932 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 19:08:14.397940 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397940 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 19:08:14.398156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397945 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 19:08:14.398156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397952 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 19:08:14.398156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397957 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 19:08:14.398156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397965 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 19:08:14.398156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.397969 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 19:08:14.398717 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.398708 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 19:08:14.398717 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.398717 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 19:08:14.402023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.402009 2577 server.go:1268] "Started kubelet" Jul 07 19:08:14.402122 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.402061 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 19:08:14.402188 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.402092 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 19:08:14.402188 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.402175 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 19:08:14.402966 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.402935 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-59.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 19:08:14.402974 ip-10-0-132-59 systemd[1]: Started Kubernetes Kubelet. Jul 07 19:08:14.403141 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.403111 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 19:08:14.403239 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.403221 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 19:08:14.404252 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.404240 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 07 19:08:14.404336 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.404240 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 19:08:14.408529 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.408502 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 19:08:14.409151 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.408215 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-59.ec2.internal.18c01780cd0a6678 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-59.ec2.internal,UID:ip-10-0-132-59.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-59.ec2.internal,},FirstTimestamp:2026-07-07 19:08:14.401980024 +0000 UTC m=+0.337605375,LastTimestamp:2026-07-07 19:08:14.401980024 +0000 UTC m=+0.337605375,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-59.ec2.internal,}" Jul 07 19:08:14.412992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.412977 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 19:08:14.412992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.412988 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 19:08:14.413134 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.413105 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:14.413932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.413624 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 19:08:14.413932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.413640 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 19:08:14.413932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.413746 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 19:08:14.413932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.413784 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 07 19:08:14.413932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.413794 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 07 19:08:14.414193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.414179 2577 factory.go:55] Registering systemd factory Jul 07 19:08:14.414261 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.414196 2577 factory.go:223] Registration of the systemd container factory successfully Jul 07 19:08:14.414468 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.414434 2577 factory.go:153] Registering CRI-O factory Jul 07 19:08:14.414616 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.414472 2577 factory.go:223] Registration of the crio container factory successfully Jul 07 19:08:14.414616 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.414565 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 19:08:14.414709 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.414621 2577 factory.go:103] Registering Raw factory Jul 07 19:08:14.414709 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.414640 2577 manager.go:1196] Started watching for new ooms in manager Jul 07 19:08:14.415911 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.415897 2577 manager.go:349] Starting recovery of all containers Jul 07 19:08:14.425023 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.424837 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 19:08:14.425023 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.424955 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 19:08:14.427359 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.427342 2577 manager.go:354] Recovery completed Jul 07 19:08:14.431276 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.431265 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:08:14.433112 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.433092 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:08:14.433185 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.433120 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:08:14.433185 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.433130 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:08:14.433667 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.433640 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 19:08:14.433667 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.433648 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:08:14.433667 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.433654 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 19:08:14.433844 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.433673 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:08:14.434279 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.434242 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:08:14.434279 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.434269 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:08:14.434279 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.434281 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:08:14.434961 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.434949 2577 policy_none.go:49] "None policy: Start" Jul 07 19:08:14.435027 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.434968 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 19:08:14.435027 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.434981 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 19:08:14.436333 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.436321 2577 policy_none.go:47] "Start" Jul 07 19:08:14.436433 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.436418 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l56zx" Jul 07 19:08:14.446884 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.446803 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-59.ec2.internal.18c01780cee56197 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-59.ec2.internal,UID:ip-10-0-132-59.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-59.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-59.ec2.internal,},FirstTimestamp:2026-07-07 19:08:14.433108375 +0000 UTC m=+0.368733725,LastTimestamp:2026-07-07 19:08:14.433108375 +0000 UTC m=+0.368733725,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-59.ec2.internal,}" Jul 07 19:08:14.450387 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.450370 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l56zx" Jul 07 19:08:14.481392 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.481378 2577 manager.go:337] "Starting Device Plugin manager" Jul 07 19:08:14.481485 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.481413 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 19:08:14.481485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.481423 2577 server.go:87] "Starting device plugin registration server" Jul 07 19:08:14.481669 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.481657 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 19:08:14.481744 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.481672 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 19:08:14.481803 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.481782 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 19:08:14.481855 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.481849 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 19:08:14.481928 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.481857 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 19:08:14.482327 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.482298 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 19:08:14.482418 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.482344 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:14.582171 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.582105 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:08:14.583690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.583085 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:08:14.583690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.583115 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:08:14.583690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.583128 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:08:14.583690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.583154 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.592059 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.592045 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.592121 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.592066 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-59.ec2.internal\": node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:14.595674 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.595653 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 19:08:14.597114 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.597097 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 19:08:14.597231 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.597134 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 19:08:14.597231 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.597157 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 19:08:14.597231 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.597188 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 19:08:14.599334 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.599319 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:08:14.602595 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.602579 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:14.698175 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.698147 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal"] Jul 07 19:08:14.698324 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.698249 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:08:14.699727 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.699710 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:08:14.699812 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.699745 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:08:14.699812 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.699761 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:08:14.701103 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.701088 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:08:14.701191 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.701154 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.701191 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.701181 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:08:14.701818 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.701806 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:08:14.701862 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.701833 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:08:14.701862 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.701847 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:08:14.702501 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.702484 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:08:14.702586 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.702515 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:08:14.702586 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.702548 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:08:14.702679 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.702665 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:14.703694 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.703680 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.703741 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.703707 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:08:14.706381 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.706366 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:08:14.706465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.706396 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:08:14.706465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.706409 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:08:14.717417 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.717399 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-59.ec2.internal\" not found" node="ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.722970 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.722954 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-59.ec2.internal\" not found" node="ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.803436 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.803413 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:14.814554 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.814533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51bb1ea7695c6c47bc1a2c41c4448726-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal\" (UID: \"51bb1ea7695c6c47bc1a2c41c4448726\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.814652 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.814566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51bb1ea7695c6c47bc1a2c41c4448726-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal\" (UID: \"51bb1ea7695c6c47bc1a2c41c4448726\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.814652 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.814593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36efaeb11e6e863bb0fd6bda41d8081a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-59.ec2.internal\" (UID: \"36efaeb11e6e863bb0fd6bda41d8081a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.904463 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:14.904394 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:14.915727 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.915696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51bb1ea7695c6c47bc1a2c41c4448726-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal\" (UID: \"51bb1ea7695c6c47bc1a2c41c4448726\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.915798 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.915731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51bb1ea7695c6c47bc1a2c41c4448726-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal\" (UID: \"51bb1ea7695c6c47bc1a2c41c4448726\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.915798 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.915749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36efaeb11e6e863bb0fd6bda41d8081a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-59.ec2.internal\" (UID: \"36efaeb11e6e863bb0fd6bda41d8081a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.915798 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.915779 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51bb1ea7695c6c47bc1a2c41c4448726-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal\" (UID: \"51bb1ea7695c6c47bc1a2c41c4448726\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.915798 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.915786 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51bb1ea7695c6c47bc1a2c41c4448726-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal\" (UID: \"51bb1ea7695c6c47bc1a2c41c4448726\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" Jul 07 19:08:14.915953 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:14.915824 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36efaeb11e6e863bb0fd6bda41d8081a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-59.ec2.internal\" (UID: \"36efaeb11e6e863bb0fd6bda41d8081a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal" Jul 07 19:08:15.004902 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.004858 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.021237 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.021215 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" Jul 07 19:08:15.027925 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.027908 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal" Jul 07 19:08:15.105831 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.105789 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.206336 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.206275 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.306843 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.306809 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.352268 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.352238 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 19:08:15.352799 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.352395 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 19:08:15.408018 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.407991 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.453061 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.453021 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 19:03:14 +0000 UTC" deadline="2028-04-17 23:19:41.261823804 +0000 UTC" Jul 07 19:08:15.453165 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.453060 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15604h11m25.808769279s" Jul 07 19:08:15.508645 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.508617 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.547326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.547299 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:08:15.548696 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.548678 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:08:15.548797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.548709 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:08:15.548797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.548719 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:08:15.580008 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:15.579977 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36efaeb11e6e863bb0fd6bda41d8081a.slice/crio-cea2c8f96aa01b0de2ba1c6d524b406549a83d45da6e558d1b2aad21e5d98ecd WatchSource:0}: Error finding container cea2c8f96aa01b0de2ba1c6d524b406549a83d45da6e558d1b2aad21e5d98ecd: Status 404 returned error can't find the container with id cea2c8f96aa01b0de2ba1c6d524b406549a83d45da6e558d1b2aad21e5d98ecd Jul 07 19:08:15.580410 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:15.580389 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bb1ea7695c6c47bc1a2c41c4448726.slice/crio-41096100d306d218796106a123efe7ef2271a72475308e5337cc37f86db7f66d WatchSource:0}: Error finding container 41096100d306d218796106a123efe7ef2271a72475308e5337cc37f86db7f66d: Status 404 returned error can't find the container with id 41096100d306d218796106a123efe7ef2271a72475308e5337cc37f86db7f66d Jul 07 19:08:15.585407 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.585386 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:08:15.600454 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.600416 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal" event={"ID":"36efaeb11e6e863bb0fd6bda41d8081a","Type":"ContainerStarted","Data":"cea2c8f96aa01b0de2ba1c6d524b406549a83d45da6e558d1b2aad21e5d98ecd"} Jul 07 19:08:15.601551 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.601531 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" event={"ID":"51bb1ea7695c6c47bc1a2c41c4448726","Type":"ContainerStarted","Data":"41096100d306d218796106a123efe7ef2271a72475308e5337cc37f86db7f66d"} Jul 07 19:08:15.609182 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.609163 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.711803 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.711771 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.802096 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.802019 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:08:15.812224 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.812195 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.912480 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:15.912444 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-59.ec2.internal\" not found" Jul 07 19:08:15.975254 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.975225 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:08:15.989115 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:15.989088 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:08:16.014002 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.013973 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" Jul 07 19:08:16.027455 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.027432 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:08:16.028510 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.028497 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal" Jul 07 19:08:16.040165 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.040146 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:08:16.335099 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.335065 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:08:16.392505 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.392470 2577 apiserver.go:52] "Watching apiserver" Jul 07 19:08:16.401402 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.401377 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 19:08:16.401884 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.401831 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-6ljzq","openshift-dns/node-resolver-pw2dn","openshift-multus/multus-wbn2c","openshift-network-diagnostics/network-check-target-wv6r9","openshift-image-registry/node-ca-snngx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal","openshift-multus/multus-additional-cni-plugins-j6rmz","openshift-multus/network-metrics-daemon-9jp8h","openshift-network-operator/iptables-alerter-bdk8t","openshift-ovn-kubernetes/ovnkube-node-nhfhb","kube-system/global-pull-secret-syncer-5m825","kube-system/konnectivity-agent-87p9q","kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn"] Jul 07 19:08:16.403394 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.403362 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.404739 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.404713 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.406415 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.406393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.406542 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.406461 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 19:08:16.406810 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.406793 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sw7c8\"" Jul 07 19:08:16.406981 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.406966 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:08:16.407108 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.407092 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 19:08:16.407201 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.407182 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 19:08:16.407948 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.407432 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 19:08:16.407948 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.407565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:16.407948 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.407638 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:16.407948 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.407812 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5sljq\"" Jul 07 19:08:16.408984 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.408965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.409094 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.408972 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 19:08:16.410343 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.409849 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 19:08:16.410343 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.409983 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 19:08:16.410343 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.410214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.410343 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.410251 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7xzxd\"" Jul 07 19:08:16.411824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.411606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:16.411824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.411651 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tqfjj\"" Jul 07 19:08:16.411824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.411659 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 19:08:16.411824 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.411700 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:16.412060 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.411875 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 19:08:16.412475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.412454 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 19:08:16.412655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.412640 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 19:08:16.412817 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.412803 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7tkb9\"" Jul 07 19:08:16.412885 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.412865 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:08:16.413007 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.412985 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 19:08:16.413230 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.413206 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 19:08:16.414045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.413424 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 19:08:16.414165 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.414150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.416240 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.416209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.417655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.417634 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 19:08:16.417990 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.417942 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 19:08:16.417990 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.417983 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 19:08:16.418160 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.418053 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:16.418226 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.418177 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:16.418283 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.418236 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vwxl7\"" Jul 07 19:08:16.418979 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.418869 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 19:08:16.419253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.419187 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v4kk8\"" Jul 07 19:08:16.420889 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.420867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:16.421231 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.421211 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 19:08:16.421600 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.421583 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 19:08:16.421662 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.421593 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 19:08:16.422809 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.422789 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 19:08:16.423250 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.423232 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 19:08:16.423988 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.423767 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 19:08:16.424180 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424158 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-cni-bin\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.424288 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-modprobe-d\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.424288 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b086e7b-d533-4209-8e64-993aea7b9eb6-tmp\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.424288 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-system-cni-dir\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a-serviceca\") pod \"node-ca-snngx\" (UID: \"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a\") " pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-kubelet\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-etc-openvswitch\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-kubernetes\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-sysctl-conf\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-systemd\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-sys\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/32b09ed2-5064-4548-a956-0d4760e2ba67-env-overrides\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwx8\" (UniqueName: \"kubernetes.io/projected/2c902040-dd95-4b77-912d-2f2264683405-kube-api-access-4fwx8\") pod \"node-resolver-pw2dn\" (UID: \"2c902040-dd95-4b77-912d-2f2264683405\") " pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-registration-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.424644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-var-lib-kubelet\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.425447 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0924eb3-45ee-4aae-870a-ccd46c073700-iptables-alerter-script\") pod \"iptables-alerter-bdk8t\" (UID: \"d0924eb3-45ee-4aae-870a-ccd46c073700\") " pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.425447 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a-host\") pod \"node-ca-snngx\" (UID: \"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a\") " pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.425447 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.424732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-sysctl-d\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.425447 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425355 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-host\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.425447 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-tuned\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.425447 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.425447 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-cnibin\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425470 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/32b09ed2-5064-4548-a956-0d4760e2ba67-ovn-node-metrics-cert\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-sys-fs\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5nq\" (UniqueName: \"kubernetes.io/projected/3b086e7b-d533-4209-8e64-993aea7b9eb6-kube-api-access-bl5nq\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-run-netns\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425581 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdgs9\" (UniqueName: \"kubernetes.io/projected/32b09ed2-5064-4548-a956-0d4760e2ba67-kube-api-access-jdgs9\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-systemd-units\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-node-log\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-log-socket\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.425759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425756 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6dk4\" (UniqueName: \"kubernetes.io/projected/a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a-kube-api-access-n6dk4\") pod \"node-ca-snngx\" (UID: \"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a\") " pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.426209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/32b09ed2-5064-4548-a956-0d4760e2ba67-ovnkube-config\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.426209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kqkk\" (UniqueName: \"kubernetes.io/projected/00c7570a-9234-403e-9bf6-7a3074d0be5e-kube-api-access-4kqkk\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.426209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-cni-binary-copy\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.426209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.426209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-run-systemd\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.426209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/32b09ed2-5064-4548-a956-0d4760e2ba67-ovnkube-script-lib\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.426209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0924eb3-45ee-4aae-870a-ccd46c073700-host-slash\") pod \"iptables-alerter-bdk8t\" (UID: \"d0924eb3-45ee-4aae-870a-ccd46c073700\") " pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.426209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr6g5\" (UniqueName: \"kubernetes.io/projected/d0924eb3-45ee-4aae-870a-ccd46c073700-kube-api-access-nr6g5\") pod \"iptables-alerter-bdk8t\" (UID: \"d0924eb3-45ee-4aae-870a-ccd46c073700\") " pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.426209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.425655 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w9s8r\"" Jul 07 19:08:16.426641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426296 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dmc\" (UniqueName: \"kubernetes.io/projected/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-kube-api-access-n5dmc\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.426641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:16.426641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426346 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 19:08:16.426641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-run-openvswitch\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.426641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-run-ovn\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.426641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-run-ovn-kubernetes\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.426641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-run\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.426641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426616 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwfm\" (UniqueName: \"kubernetes.io/projected/969a6be0-5242-4a03-9c44-befc4165567b-kube-api-access-fmwfm\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-cni-netd\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-sysconfig\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c902040-dd95-4b77-912d-2f2264683405-hosts-file\") pod \"node-resolver-pw2dn\" (UID: \"2c902040-dd95-4b77-912d-2f2264683405\") " pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-device-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-os-release\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426755 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c902040-dd95-4b77-912d-2f2264683405-tmp-dir\") pod \"node-resolver-pw2dn\" (UID: \"2c902040-dd95-4b77-912d-2f2264683405\") " pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426803 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-socket-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426820 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-lib-modules\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-slash\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.427256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.426847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-var-lib-openvswitch\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.427993 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.427839 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 19:08:16.428370 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.428351 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bpcqm\"" Jul 07 19:08:16.433578 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.433559 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:08:16.456494 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.456471 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pfngz" Jul 07 19:08:16.465764 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.465744 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pfngz" Jul 07 19:08:16.515328 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.515298 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 19:08:16.527399 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-run-netns\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.527399 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdgs9\" (UniqueName: \"kubernetes.io/projected/32b09ed2-5064-4548-a956-0d4760e2ba67-kube-api-access-jdgs9\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.527565 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527470 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-run-netns\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.527565 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f4a402b-2e9f-4b07-9b1c-3b77776a6358-konnectivity-ca\") pod \"konnectivity-agent-87p9q\" (UID: \"2f4a402b-2e9f-4b07-9b1c-3b77776a6358\") " pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:16.527565 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.527666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.527666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-systemd-units\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.527666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-node-log\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.527759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-log-socket\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.527759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527710 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-node-log\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.527759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-var-lib-cni-multus\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.527759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.527759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6dk4\" (UniqueName: \"kubernetes.io/projected/a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a-kube-api-access-n6dk4\") pod \"node-ca-snngx\" (UID: \"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a\") " pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.527935 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527770 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.527935 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/32b09ed2-5064-4548-a956-0d4760e2ba67-ovnkube-config\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.527935 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-multus-cni-dir\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.527935 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-var-lib-kubelet\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.527935 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d6ebd01-5778-4838-9480-a225150350cf-multus-daemon-config\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.527935 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kqkk\" (UniqueName: \"kubernetes.io/projected/00c7570a-9234-403e-9bf6-7a3074d0be5e-kube-api-access-4kqkk\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.527935 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-cni-binary-copy\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.528182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.528182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.527995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-run-multus-certs\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.528182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-run-systemd\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.528182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/32b09ed2-5064-4548-a956-0d4760e2ba67-ovnkube-script-lib\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.528182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0924eb3-45ee-4aae-870a-ccd46c073700-host-slash\") pod \"iptables-alerter-bdk8t\" (UID: \"d0924eb3-45ee-4aae-870a-ccd46c073700\") " pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.528182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr6g5\" (UniqueName: \"kubernetes.io/projected/d0924eb3-45ee-4aae-870a-ccd46c073700-kube-api-access-nr6g5\") pod \"iptables-alerter-bdk8t\" (UID: \"d0924eb3-45ee-4aae-870a-ccd46c073700\") " pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.528182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dmc\" (UniqueName: \"kubernetes.io/projected/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-kube-api-access-n5dmc\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.528182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-run-openvswitch\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-run-ovn\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-run-ovn-kubernetes\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d6ebd01-5778-4838-9480-a225150350cf-cni-binary-copy\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528278 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-log-socket\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-run\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwfm\" (UniqueName: \"kubernetes.io/projected/969a6be0-5242-4a03-9c44-befc4165567b-kube-api-access-fmwfm\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-cni-netd\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-var-lib-cni-bin\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-sysconfig\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c902040-dd95-4b77-912d-2f2264683405-hosts-file\") pod \"node-resolver-pw2dn\" (UID: \"2c902040-dd95-4b77-912d-2f2264683405\") " pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.528473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-run-k8s-cni-cncf-io\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-device-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-os-release\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-dbus\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-hostroot\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c902040-dd95-4b77-912d-2f2264683405-tmp-dir\") pod \"node-resolver-pw2dn\" (UID: \"2c902040-dd95-4b77-912d-2f2264683405\") " pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-etc-kubernetes\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd67h\" (UniqueName: \"kubernetes.io/projected/8d6ebd01-5778-4838-9480-a225150350cf-kube-api-access-qd67h\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f4a402b-2e9f-4b07-9b1c-3b77776a6358-agent-certs\") pod \"konnectivity-agent-87p9q\" (UID: \"2f4a402b-2e9f-4b07-9b1c-3b77776a6358\") " pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-socket-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528738 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-lib-modules\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528785 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-slash\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-var-lib-openvswitch\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-cni-bin\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-modprobe-d\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.529110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b086e7b-d533-4209-8e64-993aea7b9eb6-tmp\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-system-cni-dir\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a-serviceca\") pod \"node-ca-snngx\" (UID: \"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a\") " pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-kubelet\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-etc-openvswitch\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-kubelet-config\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-kubernetes\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-sysctl-conf\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529153 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-systemd\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-sys\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/32b09ed2-5064-4548-a956-0d4760e2ba67-env-overrides\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.528238 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/32b09ed2-5064-4548-a956-0d4760e2ba67-ovnkube-config\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529343 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-cni-netd\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529374 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-sys\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-sysconfig\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529445 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c902040-dd95-4b77-912d-2f2264683405-hosts-file\") pod \"node-resolver-pw2dn\" (UID: \"2c902040-dd95-4b77-912d-2f2264683405\") " pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwx8\" (UniqueName: \"kubernetes.io/projected/2c902040-dd95-4b77-912d-2f2264683405-kube-api-access-4fwx8\") pod \"node-resolver-pw2dn\" (UID: \"2c902040-dd95-4b77-912d-2f2264683405\") " pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-registration-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-var-lib-kubelet\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0924eb3-45ee-4aae-870a-ccd46c073700-iptables-alerter-script\") pod \"iptables-alerter-bdk8t\" (UID: \"d0924eb3-45ee-4aae-870a-ccd46c073700\") " pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a-host\") pod \"node-ca-snngx\" (UID: \"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a\") " pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529581 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-run-netns\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529612 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-system-cni-dir\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-sysctl-d\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-host\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-tuned\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-system-cni-dir\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-os-release\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-multus-socket-dir-parent\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.530923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-multus-conf-dir\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-cnibin\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/32b09ed2-5064-4548-a956-0d4760e2ba67-ovn-node-metrics-cert\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-cnibin\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-sys-fs\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5nq\" (UniqueName: \"kubernetes.io/projected/3b086e7b-d533-4209-8e64-993aea7b9eb6-kube-api-access-bl5nq\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529895 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-kubelet\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.529942 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-etc-openvswitch\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530025 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a-serviceca\") pod \"node-ca-snngx\" (UID: \"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a\") " pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530084 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-run-ovn\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530093 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-sysctl-conf\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530254 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-run-systemd\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530403 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/32b09ed2-5064-4548-a956-0d4760e2ba67-env-overrides\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-registration-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530503 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-var-lib-kubelet\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-systemd\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530706 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.531824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530751 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-var-lib-openvswitch\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530794 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-run-ovn-kubernetes\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530831 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-cni-bin\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530905 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-modprobe-d\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.530926 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/32b09ed2-5064-4548-a956-0d4760e2ba67-ovnkube-script-lib\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531018 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-sys-fs\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531040 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0924eb3-45ee-4aae-870a-ccd46c073700-iptables-alerter-script\") pod \"iptables-alerter-bdk8t\" (UID: \"d0924eb3-45ee-4aae-870a-ccd46c073700\") " pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531113 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a-host\") pod \"node-ca-snngx\" (UID: \"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a\") " pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-run\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531193 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-sysctl-d\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531234 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-host\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531267 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-cnibin\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-device-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531332 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-kubernetes\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531521 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-os-release\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531570 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c902040-dd95-4b77-912d-2f2264683405-tmp-dir\") pod \"node-resolver-pw2dn\" (UID: \"2c902040-dd95-4b77-912d-2f2264683405\") " pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.531663 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:16.532723 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.531723 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs podName:969a6be0-5242-4a03-9c44-befc4165567b nodeName:}" failed. No retries permitted until 2026-07-07 19:08:17.031708537 +0000 UTC m=+2.967333878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs") pod "network-metrics-daemon-9jp8h" (UID: "969a6be0-5242-4a03-9c44-befc4165567b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:16.533736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-systemd-units\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.533736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531840 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b086e7b-d533-4209-8e64-993aea7b9eb6-lib-modules\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.533736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.531918 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00c7570a-9234-403e-9bf6-7a3074d0be5e-socket-dir\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.533736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.532123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-run-openvswitch\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.533736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.532164 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.533736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.532193 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-cni-binary-copy\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.533736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.532209 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32b09ed2-5064-4548-a956-0d4760e2ba67-host-slash\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.533736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.532213 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.533736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.532263 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0924eb3-45ee-4aae-870a-ccd46c073700-host-slash\") pod \"iptables-alerter-bdk8t\" (UID: \"d0924eb3-45ee-4aae-870a-ccd46c073700\") " pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.534210 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.533766 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b086e7b-d533-4209-8e64-993aea7b9eb6-tmp\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.534210 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.534070 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b086e7b-d533-4209-8e64-993aea7b9eb6-etc-tuned\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.534210 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.534123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/32b09ed2-5064-4548-a956-0d4760e2ba67-ovn-node-metrics-cert\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.537294 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.537268 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdgs9\" (UniqueName: \"kubernetes.io/projected/32b09ed2-5064-4548-a956-0d4760e2ba67-kube-api-access-jdgs9\") pod \"ovnkube-node-nhfhb\" (UID: \"32b09ed2-5064-4548-a956-0d4760e2ba67\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.537615 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.537590 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwfm\" (UniqueName: \"kubernetes.io/projected/969a6be0-5242-4a03-9c44-befc4165567b-kube-api-access-fmwfm\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:16.537733 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.537711 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6dk4\" (UniqueName: \"kubernetes.io/projected/a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a-kube-api-access-n6dk4\") pod \"node-ca-snngx\" (UID: \"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a\") " pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.539772 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.539593 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:08:16.539772 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.539616 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:08:16.539772 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.539629 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tl for pod openshift-network-diagnostics/network-check-target-wv6r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:16.539772 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.539705 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl podName:9c4370b6-71fb-4592-8bf4-f9060e2448c5 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:17.03967871 +0000 UTC m=+2.975304066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tl" (UniqueName: "kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl") pod "network-check-target-wv6r9" (UID: "9c4370b6-71fb-4592-8bf4-f9060e2448c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:16.540684 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.540663 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5nq\" (UniqueName: \"kubernetes.io/projected/3b086e7b-d533-4209-8e64-993aea7b9eb6-kube-api-access-bl5nq\") pod \"tuned-6ljzq\" (UID: \"3b086e7b-d533-4209-8e64-993aea7b9eb6\") " pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.542577 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.542552 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwx8\" (UniqueName: \"kubernetes.io/projected/2c902040-dd95-4b77-912d-2f2264683405-kube-api-access-4fwx8\") pod \"node-resolver-pw2dn\" (UID: \"2c902040-dd95-4b77-912d-2f2264683405\") " pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.542709 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.542686 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kqkk\" (UniqueName: \"kubernetes.io/projected/00c7570a-9234-403e-9bf6-7a3074d0be5e-kube-api-access-4kqkk\") pod \"aws-ebs-csi-driver-node-59lkn\" (UID: \"00c7570a-9234-403e-9bf6-7a3074d0be5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.546765 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.546740 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr6g5\" (UniqueName: \"kubernetes.io/projected/d0924eb3-45ee-4aae-870a-ccd46c073700-kube-api-access-nr6g5\") pod \"iptables-alerter-bdk8t\" (UID: \"d0924eb3-45ee-4aae-870a-ccd46c073700\") " pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.547550 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.547524 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dmc\" (UniqueName: \"kubernetes.io/projected/b4982ea3-c0c5-4478-be98-5ca0f95b3f40-kube-api-access-n5dmc\") pod \"multus-additional-cni-plugins-j6rmz\" (UID: \"b4982ea3-c0c5-4478-be98-5ca0f95b3f40\") " pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.631144 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-system-cni-dir\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631144 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-os-release\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631144 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-multus-socket-dir-parent\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-multus-conf-dir\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-cnibin\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f4a402b-2e9f-4b07-9b1c-3b77776a6358-konnectivity-ca\") pod \"konnectivity-agent-87p9q\" (UID: \"2f4a402b-2e9f-4b07-9b1c-3b77776a6358\") " pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-var-lib-cni-multus\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-multus-cni-dir\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631248 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-multus-conf-dir\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-var-lib-kubelet\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d6ebd01-5778-4838-9480-a225150350cf-multus-daemon-config\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-run-multus-certs\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631322 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-os-release\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d6ebd01-5778-4838-9480-a225150350cf-cni-binary-copy\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631372 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-multus-socket-dir-parent\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631391 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-var-lib-cni-bin\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.631445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-run-k8s-cni-cncf-io\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-dbus\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-hostroot\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-etc-kubernetes\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd67h\" (UniqueName: \"kubernetes.io/projected/8d6ebd01-5778-4838-9480-a225150350cf-kube-api-access-qd67h\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631524 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-multus-cni-dir\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-cnibin\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631760 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-var-lib-kubelet\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632206 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f4a402b-2e9f-4b07-9b1c-3b77776a6358-konnectivity-ca\") pod \"konnectivity-agent-87p9q\" (UID: \"2f4a402b-2e9f-4b07-9b1c-3b77776a6358\") " pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632278 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-var-lib-cni-multus\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632281 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d6ebd01-5778-4838-9480-a225150350cf-multus-daemon-config\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.631212 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-system-cni-dir\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.632376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632341 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-run-k8s-cni-cncf-io\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632385 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-run-multus-certs\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-dbus\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632481 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-hostroot\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632508 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-etc-kubernetes\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f4a402b-2e9f-4b07-9b1c-3b77776a6358-agent-certs\") pod \"konnectivity-agent-87p9q\" (UID: \"2f4a402b-2e9f-4b07-9b1c-3b77776a6358\") " pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-kubelet-config\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-run-netns\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632665 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-run-netns\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.633014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.632979 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d6ebd01-5778-4838-9480-a225150350cf-cni-binary-copy\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.633599 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.633013 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d6ebd01-5778-4838-9480-a225150350cf-host-var-lib-cni-bin\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.633599 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.633111 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:16.633599 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.633131 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-kubelet-config\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:16.633599 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:16.633170 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret podName:80d3ae4b-a4d1-4f35-8815-61d9a05251ba nodeName:}" failed. No retries permitted until 2026-07-07 19:08:17.13314975 +0000 UTC m=+3.068775093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret") pod "global-pull-secret-syncer-5m825" (UID: "80d3ae4b-a4d1-4f35-8815-61d9a05251ba") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:16.635536 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.635515 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f4a402b-2e9f-4b07-9b1c-3b77776a6358-agent-certs\") pod \"konnectivity-agent-87p9q\" (UID: \"2f4a402b-2e9f-4b07-9b1c-3b77776a6358\") " pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:16.642245 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.642217 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd67h\" (UniqueName: \"kubernetes.io/projected/8d6ebd01-5778-4838-9480-a225150350cf-kube-api-access-qd67h\") pod \"multus-wbn2c\" (UID: \"8d6ebd01-5778-4838-9480-a225150350cf\") " pod="openshift-multus/multus-wbn2c" Jul 07 19:08:16.718151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.718122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdk8t" Jul 07 19:08:16.728615 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.728590 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pw2dn" Jul 07 19:08:16.739799 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.739780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" Jul 07 19:08:16.745787 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.745767 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" Jul 07 19:08:16.754783 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.754760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" Jul 07 19:08:16.771547 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.771529 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-snngx" Jul 07 19:08:16.779634 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.779614 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:16.787491 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.787474 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:16.795300 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:16.795283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbn2c" Jul 07 19:08:17.036374 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:17.036346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:17.036525 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:17.036497 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:17.036583 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:17.036570 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs podName:969a6be0-5242-4a03-9c44-befc4165567b nodeName:}" failed. No retries permitted until 2026-07-07 19:08:18.03654626 +0000 UTC m=+3.972171601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs") pod "network-metrics-daemon-9jp8h" (UID: "969a6be0-5242-4a03-9c44-befc4165567b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:17.136931 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:17.136834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:17.136931 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:17.136884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:17.137166 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:17.136999 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:17.137166 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:17.137020 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:08:17.137166 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:17.137073 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret podName:80d3ae4b-a4d1-4f35-8815-61d9a05251ba nodeName:}" failed. No retries permitted until 2026-07-07 19:08:18.137048967 +0000 UTC m=+4.072674307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret") pod "global-pull-secret-syncer-5m825" (UID: "80d3ae4b-a4d1-4f35-8815-61d9a05251ba") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:17.137166 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:17.137078 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:08:17.137166 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:17.137092 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tl for pod openshift-network-diagnostics/network-check-target-wv6r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:17.137382 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:17.137169 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl podName:9c4370b6-71fb-4592-8bf4-f9060e2448c5 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:18.13715143 +0000 UTC m=+4.072776772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tl" (UniqueName: "kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl") pod "network-check-target-wv6r9" (UID: "9c4370b6-71fb-4592-8bf4-f9060e2448c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:17.468244 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:17.468172 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 19:03:16 +0000 UTC" deadline="2028-03-05 03:09:34.662413197 +0000 UTC" Jul 07 19:08:17.468244 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:17.468200 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14552h1m17.19421635s" Jul 07 19:08:17.782718 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:17.782673 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c7570a_9234_403e_9bf6_7a3074d0be5e.slice/crio-066003fbfa2a9d5c155532c0285b586dcef2c27fdb365a195b61342ed168c738 WatchSource:0}: Error finding container 066003fbfa2a9d5c155532c0285b586dcef2c27fdb365a195b61342ed168c738: Status 404 returned error can't find the container with id 066003fbfa2a9d5c155532c0285b586dcef2c27fdb365a195b61342ed168c738 Jul 07 19:08:17.784531 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:17.784465 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0924eb3_45ee_4aae_870a_ccd46c073700.slice/crio-f42209aebe17783e0545f233bed5659599ba1ef573b4f0405093f1c47faadb9f WatchSource:0}: Error finding container f42209aebe17783e0545f233bed5659599ba1ef573b4f0405093f1c47faadb9f: Status 404 returned error can't find the container with id f42209aebe17783e0545f233bed5659599ba1ef573b4f0405093f1c47faadb9f Jul 07 19:08:17.786646 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:17.786626 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4a402b_2e9f_4b07_9b1c_3b77776a6358.slice/crio-5339bf8a02177b0f0daa3ec57a6f99a75f4377f06a91322432ca1bbf70c91160 WatchSource:0}: Error finding container 5339bf8a02177b0f0daa3ec57a6f99a75f4377f06a91322432ca1bbf70c91160: Status 404 returned error can't find the container with id 5339bf8a02177b0f0daa3ec57a6f99a75f4377f06a91322432ca1bbf70c91160 Jul 07 19:08:17.787511 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:17.787489 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b09ed2_5064_4548_a956_0d4760e2ba67.slice/crio-bb8f107ab1fae42d5852cc6784d9e11abfb09d483c64741428010e71886d5864 WatchSource:0}: Error finding container bb8f107ab1fae42d5852cc6784d9e11abfb09d483c64741428010e71886d5864: Status 404 returned error can't find the container with id bb8f107ab1fae42d5852cc6784d9e11abfb09d483c64741428010e71886d5864 Jul 07 19:08:17.794872 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:08:17.794417 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6ebd01_5778_4838_9480_a225150350cf.slice/crio-b70ee6486d6cb38875f2491165d0e0610c6404a84ccddbbea5ff8c555e315f3e WatchSource:0}: Error finding container b70ee6486d6cb38875f2491165d0e0610c6404a84ccddbbea5ff8c555e315f3e: Status 404 returned error can't find the container with id b70ee6486d6cb38875f2491165d0e0610c6404a84ccddbbea5ff8c555e315f3e Jul 07 19:08:18.043768 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.043469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:18.043972 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.043659 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:18.043972 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.043848 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs podName:969a6be0-5242-4a03-9c44-befc4165567b nodeName:}" failed. No retries permitted until 2026-07-07 19:08:20.043825912 +0000 UTC m=+5.979451258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs") pod "network-metrics-daemon-9jp8h" (UID: "969a6be0-5242-4a03-9c44-befc4165567b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:18.144507 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.144478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:18.144645 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.144521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:18.144645 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.144629 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:18.144757 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.144665 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:08:18.144757 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.144684 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret podName:80d3ae4b-a4d1-4f35-8815-61d9a05251ba nodeName:}" failed. No retries permitted until 2026-07-07 19:08:20.144670676 +0000 UTC m=+6.080296016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret") pod "global-pull-secret-syncer-5m825" (UID: "80d3ae4b-a4d1-4f35-8815-61d9a05251ba") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:18.144757 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.144693 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:08:18.144757 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.144708 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tl for pod openshift-network-diagnostics/network-check-target-wv6r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:18.144916 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.144763 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl podName:9c4370b6-71fb-4592-8bf4-f9060e2448c5 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:20.144746719 +0000 UTC m=+6.080372063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tl" (UniqueName: "kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl") pod "network-check-target-wv6r9" (UID: "9c4370b6-71fb-4592-8bf4-f9060e2448c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:18.469326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.469206 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 19:03:16 +0000 UTC" deadline="2028-01-29 19:15:05.279931847 +0000 UTC" Jul 07 19:08:18.469326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.469269 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13704h6m46.810667624s" Jul 07 19:08:18.597941 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.597691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:18.597941 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.597792 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:18.598204 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.598007 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:18.598204 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.598131 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:18.598204 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.598163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:18.598368 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:18.598283 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:18.633646 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.633581 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal" event={"ID":"36efaeb11e6e863bb0fd6bda41d8081a","Type":"ContainerStarted","Data":"005cca0a6897cc852a29634469c1b94cdcb1ec9c8e3bf995d973eb55fb8e270d"} Jul 07 19:08:18.643262 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.642611 2577 generic.go:356] "Generic (PLEG): container finished" podID="51bb1ea7695c6c47bc1a2c41c4448726" containerID="f1092ac7c0f88c3329b74e8e7106a671e5c75538be9891fa8430ac8b0acd3b70" exitCode=0 Jul 07 19:08:18.643262 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.642697 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" event={"ID":"51bb1ea7695c6c47bc1a2c41c4448726","Type":"ContainerDied","Data":"f1092ac7c0f88c3329b74e8e7106a671e5c75538be9891fa8430ac8b0acd3b70"} Jul 07 19:08:18.658168 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.657625 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-59.ec2.internal" podStartSLOduration=2.657605431 podStartE2EDuration="2.657605431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:18.657028441 +0000 UTC m=+4.592653804" watchObservedRunningTime="2026-07-07 19:08:18.657605431 +0000 UTC m=+4.593230798" Jul 07 19:08:18.663303 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.663270 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" event={"ID":"b4982ea3-c0c5-4478-be98-5ca0f95b3f40","Type":"ContainerStarted","Data":"48ed337bef309f29431f0c9d0bc00ab47b199e6a3b72332943c61b6c26fbb80b"} Jul 07 19:08:18.677453 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.677382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" event={"ID":"3b086e7b-d533-4209-8e64-993aea7b9eb6","Type":"ContainerStarted","Data":"4e9cb0f851613d6487bd5b65e93aff4261733b9b1755039a4146808b20de92e1"} Jul 07 19:08:18.695534 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.695480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-snngx" event={"ID":"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a","Type":"ContainerStarted","Data":"076395aa0c8b00b75e4b67e930954a4d5d6e8ccab7b0faf07284b75d8a178f92"} Jul 07 19:08:18.713540 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.713428 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-87p9q" event={"ID":"2f4a402b-2e9f-4b07-9b1c-3b77776a6358","Type":"ContainerStarted","Data":"5339bf8a02177b0f0daa3ec57a6f99a75f4377f06a91322432ca1bbf70c91160"} Jul 07 19:08:18.733901 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.733698 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" event={"ID":"00c7570a-9234-403e-9bf6-7a3074d0be5e","Type":"ContainerStarted","Data":"066003fbfa2a9d5c155532c0285b586dcef2c27fdb365a195b61342ed168c738"} Jul 07 19:08:18.739227 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.739158 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pw2dn" event={"ID":"2c902040-dd95-4b77-912d-2f2264683405","Type":"ContainerStarted","Data":"e520feaeaf16fef0649d4ea3b0caae95a0592430e7daa264dd3e2816ba3e357d"} Jul 07 19:08:18.759907 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.759820 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbn2c" event={"ID":"8d6ebd01-5778-4838-9480-a225150350cf","Type":"ContainerStarted","Data":"b70ee6486d6cb38875f2491165d0e0610c6404a84ccddbbea5ff8c555e315f3e"} Jul 07 19:08:18.789348 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.789314 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" event={"ID":"32b09ed2-5064-4548-a956-0d4760e2ba67","Type":"ContainerStarted","Data":"bb8f107ab1fae42d5852cc6784d9e11abfb09d483c64741428010e71886d5864"} Jul 07 19:08:18.805893 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:18.805819 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdk8t" event={"ID":"d0924eb3-45ee-4aae-870a-ccd46c073700","Type":"ContainerStarted","Data":"f42209aebe17783e0545f233bed5659599ba1ef573b4f0405093f1c47faadb9f"} Jul 07 19:08:19.823713 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:19.823676 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" event={"ID":"51bb1ea7695c6c47bc1a2c41c4448726","Type":"ContainerStarted","Data":"0021389d57dadf05dc2d35fb34ae3ef6e58c5f18afbc7f3c57d5f7eb32c05ed3"} Jul 07 19:08:20.063124 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:20.063080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:20.063324 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.063233 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:20.063324 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.063302 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs podName:969a6be0-5242-4a03-9c44-befc4165567b nodeName:}" failed. No retries permitted until 2026-07-07 19:08:24.063282594 +0000 UTC m=+9.998907952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs") pod "network-metrics-daemon-9jp8h" (UID: "969a6be0-5242-4a03-9c44-befc4165567b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:20.164373 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:20.163482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:20.164373 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:20.163539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:20.164373 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.163699 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:20.164373 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.163764 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret podName:80d3ae4b-a4d1-4f35-8815-61d9a05251ba nodeName:}" failed. No retries permitted until 2026-07-07 19:08:24.163745592 +0000 UTC m=+10.099370933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret") pod "global-pull-secret-syncer-5m825" (UID: "80d3ae4b-a4d1-4f35-8815-61d9a05251ba") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:20.164373 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.164211 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:08:20.164373 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.164234 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:08:20.164373 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.164248 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tl for pod openshift-network-diagnostics/network-check-target-wv6r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:20.164373 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.164293 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl podName:9c4370b6-71fb-4592-8bf4-f9060e2448c5 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:24.164278381 +0000 UTC m=+10.099903738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tl" (UniqueName: "kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl") pod "network-check-target-wv6r9" (UID: "9c4370b6-71fb-4592-8bf4-f9060e2448c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:20.601357 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:20.600463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:20.601357 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.600606 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:20.601357 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:20.600998 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:20.601357 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.601091 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:20.601357 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:20.601147 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:20.601357 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:20.601212 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:22.600825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:22.600791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:22.601208 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:22.601025 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:22.601208 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:22.601116 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:22.601307 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:22.601281 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:22.601366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:22.601346 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:22.601447 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:22.601432 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:24.097894 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:24.097841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:24.098303 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.098006 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:24.098303 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.098079 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs podName:969a6be0-5242-4a03-9c44-befc4165567b nodeName:}" failed. No retries permitted until 2026-07-07 19:08:32.098058674 +0000 UTC m=+18.033684016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs") pod "network-metrics-daemon-9jp8h" (UID: "969a6be0-5242-4a03-9c44-befc4165567b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:24.198321 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:24.198289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:24.198492 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:24.198333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:24.198492 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.198448 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:24.198492 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.198473 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:08:24.198640 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.198495 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:08:24.198640 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.198507 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tl for pod openshift-network-diagnostics/network-check-target-wv6r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:24.198640 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.198508 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret podName:80d3ae4b-a4d1-4f35-8815-61d9a05251ba nodeName:}" failed. No retries permitted until 2026-07-07 19:08:32.198489802 +0000 UTC m=+18.134115155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret") pod "global-pull-secret-syncer-5m825" (UID: "80d3ae4b-a4d1-4f35-8815-61d9a05251ba") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:24.198640 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.198552 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl podName:9c4370b6-71fb-4592-8bf4-f9060e2448c5 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:32.198538701 +0000 UTC m=+18.134164044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tl" (UniqueName: "kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl") pod "network-check-target-wv6r9" (UID: "9c4370b6-71fb-4592-8bf4-f9060e2448c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:24.598728 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:24.598701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:24.598909 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:24.598811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:24.598909 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.598813 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:24.599025 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.598954 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:24.599025 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:24.598990 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:24.599114 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:24.599060 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:26.598366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:26.598331 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:26.598796 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:26.598342 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:26.598796 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:26.598463 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:26.598796 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:26.598486 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:26.598796 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:26.598581 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:26.598796 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:26.598673 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:28.600331 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.599584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:28.600331 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:28.599703 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:28.600331 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.600072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:28.600331 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:28.600157 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:28.600331 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.600219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:28.600331 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:28.600296 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:28.843624 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.843418 2577 generic.go:356] "Generic (PLEG): container finished" podID="b4982ea3-c0c5-4478-be98-5ca0f95b3f40" containerID="28d98b73993e4040ebf3973b6ad68c81c8d1c768fcf04ffdc5592b314c1a941b" exitCode=0 Jul 07 19:08:28.843764 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.843583 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" event={"ID":"b4982ea3-c0c5-4478-be98-5ca0f95b3f40","Type":"ContainerDied","Data":"28d98b73993e4040ebf3973b6ad68c81c8d1c768fcf04ffdc5592b314c1a941b"} Jul 07 19:08:28.846151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.846101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" event={"ID":"3b086e7b-d533-4209-8e64-993aea7b9eb6","Type":"ContainerStarted","Data":"78a949277367000b9345b488c369ce70bd581177c34f259d5707687cf19d1d73"} Jul 07 19:08:28.848117 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.848091 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-snngx" event={"ID":"a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a","Type":"ContainerStarted","Data":"ac4aeeef4f858ffb4ef68f49133a13a8f044a5f7477057e73a2de1a8600b84dd"} Jul 07 19:08:28.853592 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.853567 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-87p9q" event={"ID":"2f4a402b-2e9f-4b07-9b1c-3b77776a6358","Type":"ContainerStarted","Data":"26c2834bd55852b156e247efb2a3c55aaa0d9714ad01a5d5b9a206d046f3bb37"} Jul 07 19:08:28.855045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.855018 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" event={"ID":"00c7570a-9234-403e-9bf6-7a3074d0be5e","Type":"ContainerStarted","Data":"e3c6487d2a9d13ac962457fb1761f4ebbbbeeb2612bf4b9c029048f8db05cd1e"} Jul 07 19:08:28.856512 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.856495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pw2dn" event={"ID":"2c902040-dd95-4b77-912d-2f2264683405","Type":"ContainerStarted","Data":"0bb31ab054b68a5defa277d008f725f9701f11f16642b548a755686050ad4941"} Jul 07 19:08:28.872325 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.872282 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-59.ec2.internal" podStartSLOduration=12.872266041 podStartE2EDuration="12.872266041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:19.842566096 +0000 UTC m=+5.778191462" watchObservedRunningTime="2026-07-07 19:08:28.872266041 +0000 UTC m=+14.807891405" Jul 07 19:08:28.908942 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.908904 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6ljzq" podStartSLOduration=3.30367955 podStartE2EDuration="13.908892472s" totalImagesPullingTime="10.605212922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:28.908721282 +0000 UTC m=+14.844346633" watchObservedRunningTime="2026-07-07 19:08:28.908892472 +0000 UTC m=+14.844517823" Jul 07 19:08:28.909035 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.908984 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pw2dn" podStartSLOduration=4.331987123 podStartE2EDuration="14.908980621s" totalImagesPullingTime="10.576993498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:28.889527328 +0000 UTC m=+14.825152688" watchObservedRunningTime="2026-07-07 19:08:28.908980621 +0000 UTC m=+14.844605982" Jul 07 19:08:28.926199 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.926165 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-snngx" podStartSLOduration=3.359316332 podStartE2EDuration="13.926152946s" totalImagesPullingTime="10.566836614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:28.925931685 +0000 UTC m=+14.861557045" watchObservedRunningTime="2026-07-07 19:08:28.926152946 +0000 UTC m=+14.861778306" Jul 07 19:08:28.942640 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:28.942596 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-87p9q" podStartSLOduration=3.372026145 podStartE2EDuration="13.942579838s" totalImagesPullingTime="10.570553693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:28.942341981 +0000 UTC m=+14.877967358" watchObservedRunningTime="2026-07-07 19:08:28.942579838 +0000 UTC m=+14.878205199" Jul 07 19:08:30.597480 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:30.597441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:30.597990 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:30.597546 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:30.598091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:30.598076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:30.598172 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:30.598156 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:30.598334 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:30.598314 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:30.598460 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:30.598430 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:30.862860 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:30.862778 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdk8t" event={"ID":"d0924eb3-45ee-4aae-870a-ccd46c073700","Type":"ContainerStarted","Data":"58e2294cd135bfe30f7fe29a3e1d97c78d8ff33d46628fce70ddf64b44043d3e"} Jul 07 19:08:30.880326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:30.880254 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bdk8t" podStartSLOduration=6.27967603 podStartE2EDuration="16.880239738s" totalImagesPullingTime="10.600563708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:30.879867127 +0000 UTC m=+16.815492491" watchObservedRunningTime="2026-07-07 19:08:30.880239738 +0000 UTC m=+16.815865097" Jul 07 19:08:31.807936 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:31.807901 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:31.809256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:31.809229 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:31.865849 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:31.865813 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:31.866311 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:31.866292 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-87p9q" Jul 07 19:08:32.169220 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:32.169145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:32.169387 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.169317 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:32.169387 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.169381 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs podName:969a6be0-5242-4a03-9c44-befc4165567b nodeName:}" failed. No retries permitted until 2026-07-07 19:08:48.169365536 +0000 UTC m=+34.104990878 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs") pod "network-metrics-daemon-9jp8h" (UID: "969a6be0-5242-4a03-9c44-befc4165567b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:32.270281 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:32.270239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:32.270281 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:32.270284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:32.270510 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.270407 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:32.270510 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.270418 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:08:32.270510 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.270441 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:08:32.270510 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.270453 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tl for pod openshift-network-diagnostics/network-check-target-wv6r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:32.270510 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.270458 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret podName:80d3ae4b-a4d1-4f35-8815-61d9a05251ba nodeName:}" failed. No retries permitted until 2026-07-07 19:08:48.270445211 +0000 UTC m=+34.206070549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret") pod "global-pull-secret-syncer-5m825" (UID: "80d3ae4b-a4d1-4f35-8815-61d9a05251ba") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:32.270510 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.270496 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl podName:9c4370b6-71fb-4592-8bf4-f9060e2448c5 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:48.270481974 +0000 UTC m=+34.206107312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tl" (UniqueName: "kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl") pod "network-check-target-wv6r9" (UID: "9c4370b6-71fb-4592-8bf4-f9060e2448c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:32.598525 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:32.598116 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:32.598525 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:32.598116 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:32.598525 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:32.598243 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:32.598525 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.598237 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:32.598525 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.598321 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:32.598525 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:32.598387 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:34.598615 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:34.598533 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:34.599143 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:34.598661 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:34.599377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:34.599353 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:34.599531 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:34.599495 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:34.599636 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:34.599547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:34.599636 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:34.599616 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:36.597798 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:36.597765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:36.598364 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:36.597819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:36.598364 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:36.597846 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:36.598364 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:36.597978 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:36.598528 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:36.598371 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:36.598528 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:36.598447 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:38.597891 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:38.597830 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:38.597891 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:38.597838 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:38.598401 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:38.597837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:38.598401 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:38.597959 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:38.598401 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:38.598035 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:38.598401 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:38.598099 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:39.015147 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:39.015108 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:39.102493 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:39.102458 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:39.804955 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:39.804916 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:39.865777 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:39.865738 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:40.597778 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.597575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:40.597964 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.597604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:40.597964 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:40.597896 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:40.597964 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.597662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:40.598128 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:40.598017 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:40.598128 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:40.598071 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:40.665869 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.665842 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 19:08:40.886215 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.886042 2577 generic.go:356] "Generic (PLEG): container finished" podID="b4982ea3-c0c5-4478-be98-5ca0f95b3f40" containerID="2e8249b1296dc505c0afcc170feae7a3ba990e926678fa8e96f2abd43f42b636" exitCode=0 Jul 07 19:08:40.886546 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.886123 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" event={"ID":"b4982ea3-c0c5-4478-be98-5ca0f95b3f40","Type":"ContainerDied","Data":"2e8249b1296dc505c0afcc170feae7a3ba990e926678fa8e96f2abd43f42b636"} Jul 07 19:08:40.888539 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.888514 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" event={"ID":"00c7570a-9234-403e-9bf6-7a3074d0be5e","Type":"ContainerStarted","Data":"97446eddf5755cd0b4c340093f75dcd969891c806d870c4637cbc22dc74759b1"} Jul 07 19:08:40.890093 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.890063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbn2c" event={"ID":"8d6ebd01-5778-4838-9480-a225150350cf","Type":"ContainerStarted","Data":"ff1c7b945c72e1de04bf75d026735b88889d9eb4fc2bc66dcf6b42d054469d16"} Jul 07 19:08:40.891911 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.891890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" event={"ID":"32b09ed2-5064-4548-a956-0d4760e2ba67","Type":"ContainerStarted","Data":"ff3d405fd61d6e34ebe190ef6b7849e74eeae8529c119ad5caab9ad572228bbf"} Jul 07 19:08:40.891992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.891947 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" event={"ID":"32b09ed2-5064-4548-a956-0d4760e2ba67","Type":"ContainerStarted","Data":"fb281bb09ae5950b20ca5c1b18d805c2b2951fd057797976115e832676adb84b"} Jul 07 19:08:40.931566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:40.931535 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wbn2c" podStartSLOduration=3.203689376 podStartE2EDuration="25.931522994s" totalImagesPullingTime="22.727833618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:40.931509501 +0000 UTC m=+26.867134862" watchObservedRunningTime="2026-07-07 19:08:40.931522994 +0000 UTC m=+26.867148354" Jul 07 19:08:41.058828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.058797 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:41.159200 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.159162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:41.258774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.258652 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:41.359786 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.359756 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:41.496150 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.496065 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T19:08:40.665866288Z","UUID":"349babc2-9ec3-4faa-9bf4-b8c81a09936b","Handler":null,"Name":"","Endpoint":""} Jul 07 19:08:41.497492 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.497477 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 19:08:41.497585 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.497509 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 19:08:41.896854 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.896784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" event={"ID":"32b09ed2-5064-4548-a956-0d4760e2ba67","Type":"ContainerStarted","Data":"3b34633916bc204987c69f61b9f1d7a2be4d698bbf5e597c35add78fb09ee7b3"} Jul 07 19:08:41.897269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.896891 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" event={"ID":"32b09ed2-5064-4548-a956-0d4760e2ba67","Type":"ContainerStarted","Data":"0dfc0cd2b921a22d5a10af086906cb5612f1a6215744c7a61a5c90d1be22ec88"} Jul 07 19:08:41.897269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.896928 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" event={"ID":"32b09ed2-5064-4548-a956-0d4760e2ba67","Type":"ContainerStarted","Data":"7ef350cb5e18e9d1a5184d183b0a7d27ce2676e0236c6b9922fe2f0a48deb7f8"} Jul 07 19:08:41.897269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:41.896955 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" event={"ID":"32b09ed2-5064-4548-a956-0d4760e2ba67","Type":"ContainerStarted","Data":"0a5c12950d76eed56dbd947644b81dd67b0363bf174e1994a2cfe30f58c5625a"} Jul 07 19:08:42.038521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.038490 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:42.127650 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.127616 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:42.407269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.407238 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:42.508278 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.508253 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:42.597497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.597466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:42.597655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.597469 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:42.597655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.597572 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:42.597655 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:42.597571 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:42.597776 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:42.597659 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:42.597776 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:42.597693 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:42.845709 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.845637 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:42.900719 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.900693 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" event={"ID":"00c7570a-9234-403e-9bf6-7a3074d0be5e","Type":"ContainerStarted","Data":"f136a45d165a05bbaa3e13a1facefbcd4353010fc7faac4cfb9c442c66a2bb94"} Jul 07 19:08:42.902745 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.902723 2577 generic.go:356] "Generic (PLEG): container finished" podID="b4982ea3-c0c5-4478-be98-5ca0f95b3f40" containerID="8bb66f3349ddfc5685467c72416d8788ddcea74546961d06d3a9f2fc5879c72c" exitCode=0 Jul 07 19:08:42.902847 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.902752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" event={"ID":"b4982ea3-c0c5-4478-be98-5ca0f95b3f40","Type":"ContainerDied","Data":"8bb66f3349ddfc5685467c72416d8788ddcea74546961d06d3a9f2fc5879c72c"} Jul 07 19:08:42.920946 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:42.920907 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59lkn" podStartSLOduration=4.885382845 podStartE2EDuration="28.920894966s" totalImagesPullingTime="24.035512121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:42.91960005 +0000 UTC m=+28.855225409" watchObservedRunningTime="2026-07-07 19:08:42.920894966 +0000 UTC m=+28.856520320" Jul 07 19:08:43.908326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:43.908291 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" event={"ID":"32b09ed2-5064-4548-a956-0d4760e2ba67","Type":"ContainerStarted","Data":"6ac92ca2b27f07e64aaf8bd9cbd9d44ef486a622554b12736da38d96d7ff66ab"} Jul 07 19:08:44.598657 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:44.598622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:44.598825 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:44.598720 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:44.598825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:44.598816 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:44.598948 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:44.598928 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:44.598992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:44.598981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:44.599070 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:44.599054 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:44.912395 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:44.912303 2577 generic.go:356] "Generic (PLEG): container finished" podID="b4982ea3-c0c5-4478-be98-5ca0f95b3f40" containerID="a744971c03c13e90e780821f67a69f847217cfac7de1627e987d0a5dbc345f6c" exitCode=0 Jul 07 19:08:44.912395 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:44.912346 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" event={"ID":"b4982ea3-c0c5-4478-be98-5ca0f95b3f40","Type":"ContainerDied","Data":"a744971c03c13e90e780821f67a69f847217cfac7de1627e987d0a5dbc345f6c"} Jul 07 19:08:45.918075 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:45.917889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" event={"ID":"32b09ed2-5064-4548-a956-0d4760e2ba67","Type":"ContainerStarted","Data":"6479f5d0ad926744b4e0c6f6224af383d9331e3bd586a6f8e5ab99770d9c7c95"} Jul 07 19:08:45.918526 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:45.918147 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:45.918526 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:45.918193 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:45.933949 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:45.933924 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:45.945329 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:45.945295 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" podStartSLOduration=8.186630496 podStartE2EDuration="30.945283705s" totalImagesPullingTime="22.758653209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:45.943655951 +0000 UTC m=+31.879281310" watchObservedRunningTime="2026-07-07 19:08:45.945283705 +0000 UTC m=+31.880909090" Jul 07 19:08:46.597522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:46.597490 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:46.597703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:46.597499 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:46.597703 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:46.597634 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:46.597703 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:46.597679 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:46.597822 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:46.597713 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:46.597822 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:46.597790 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:46.675433 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:46.675390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:46.774715 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:46.774684 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:46.921889 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:46.921297 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:46.938471 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:46.938442 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:08:47.512011 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:47.511971 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9jp8h"] Jul 07 19:08:47.512178 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:47.512103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:47.512249 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:47.512220 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:47.512824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:47.512799 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5m825"] Jul 07 19:08:47.512912 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:47.512903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:47.513020 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:47.512998 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:47.519484 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:47.519456 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wv6r9"] Jul 07 19:08:47.519597 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:47.519582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:47.519695 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:47.519663 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:47.698323 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:47.698250 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:47.790619 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:47.790566 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:48.187503 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:48.187474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:48.187919 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:48.187604 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:48.187919 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:48.187656 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs podName:969a6be0-5242-4a03-9c44-befc4165567b nodeName:}" failed. No retries permitted until 2026-07-07 19:09:20.187639829 +0000 UTC m=+66.123265181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs") pod "network-metrics-daemon-9jp8h" (UID: "969a6be0-5242-4a03-9c44-befc4165567b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:08:48.288855 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:48.288811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:48.289020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:48.288862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:48.289020 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:48.288963 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:08:48.289020 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:48.288981 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:48.289020 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:48.288984 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:08:48.289020 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:48.289000 2577 projected.go:196] Error preparing data for projected volume kube-api-access-xg6tl for pod openshift-network-diagnostics/network-check-target-wv6r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:48.289199 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:48.289041 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret podName:80d3ae4b-a4d1-4f35-8815-61d9a05251ba nodeName:}" failed. No retries permitted until 2026-07-07 19:09:20.289022988 +0000 UTC m=+66.224648343 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret") pod "global-pull-secret-syncer-5m825" (UID: "80d3ae4b-a4d1-4f35-8815-61d9a05251ba") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:08:48.289199 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:48.289058 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl podName:9c4370b6-71fb-4592-8bf4-f9060e2448c5 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:20.289049043 +0000 UTC m=+66.224674385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xg6tl" (UniqueName: "kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl") pod "network-check-target-wv6r9" (UID: "9c4370b6-71fb-4592-8bf4-f9060e2448c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:08:48.476415 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:48.476334 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:48.548828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:48.548797 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:48.620438 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:48.620406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:48.715851 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:48.715823 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:48.785164 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:48.785133 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:48.969774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:48.969725 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:49.068934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:49.068841 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:49.372771 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:49.372686 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:49.471892 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:49.471843 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:49.598292 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:49.598255 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:49.598434 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:49.598297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:49.598434 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:49.598271 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:49.598434 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:49.598390 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:49.598537 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:49.598495 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:49.598604 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:49.598584 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:51.062808 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:51.062774 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:51.162990 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:51.162963 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:51.264666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:51.264491 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pw2dn_2c902040-dd95-4b77-912d-2f2264683405/dns-node-resolver/0.log" Jul 07 19:08:51.597820 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:51.597751 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:51.597949 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:51.597850 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:51.597949 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:51.597857 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:51.597949 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:51.597896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:51.598079 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:51.598000 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:51.598079 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:51.598061 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:51.934760 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:51.934689 2577 generic.go:356] "Generic (PLEG): container finished" podID="b4982ea3-c0c5-4478-be98-5ca0f95b3f40" containerID="b7374b69c92999fbc51b4524c566129f44e2af6d3bd88fea22c9b919a8b514bd" exitCode=0 Jul 07 19:08:51.934760 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:51.934736 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" event={"ID":"b4982ea3-c0c5-4478-be98-5ca0f95b3f40","Type":"ContainerDied","Data":"b7374b69c92999fbc51b4524c566129f44e2af6d3bd88fea22c9b919a8b514bd"} Jul 07 19:08:52.265437 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:52.265412 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-snngx_a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a/node-ca/0.log" Jul 07 19:08:52.289746 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:52.289723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:52.939737 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:52.939705 2577 generic.go:356] "Generic (PLEG): container finished" podID="b4982ea3-c0c5-4478-be98-5ca0f95b3f40" containerID="faa4583ea788764c0a3ab6321139e79e4fa0ae1f985294a7ea741b0198ba76cf" exitCode=0 Jul 07 19:08:52.939938 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:52.939755 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" event={"ID":"b4982ea3-c0c5-4478-be98-5ca0f95b3f40","Type":"ContainerDied","Data":"faa4583ea788764c0a3ab6321139e79e4fa0ae1f985294a7ea741b0198ba76cf"} Jul 07 19:08:53.597886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:53.597843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:53.598303 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:53.597843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:53.598303 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:53.597955 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:53.598303 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:53.597843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:53.598303 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:53.598024 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:53.598303 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:53.598100 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:53.945259 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:53.945172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" event={"ID":"b4982ea3-c0c5-4478-be98-5ca0f95b3f40","Type":"ContainerStarted","Data":"fde1ee95a1ffac36d9fb88caf8d8da3b92d57d5b9e53bc5108a33f1972b45eb3"} Jul 07 19:08:54.800233 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:54.800197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:54.860091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:54.860066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:55.184263 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:55.184183 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:55.284532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:55.284505 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:55.451078 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:55.451003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:55.516501 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:55.516472 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:55.597586 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:55.597562 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:55.597718 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:55.597659 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:55.597718 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:55.597667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:55.597788 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:55.597663 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:55.597788 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:55.597743 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:55.597850 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:55.597817 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:55.658927 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:55.658901 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:55.899788 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:55.899759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:56.038396 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:56.038359 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:56.892810 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:56.892778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:57.089094 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:57.089067 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:57.286229 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:57.286200 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:57.598105 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:57.598028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:57.598105 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:57.598063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:57.598279 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:57.598143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:57.598279 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:57.598157 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:57.598279 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:57.598209 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:57.598279 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:57.598265 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:57.681934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:57.681896 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:58.285156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:58.285121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:58.775437 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:58.775406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:58.891377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:58.891344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:59.088291 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:59.088223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:59.190398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:59.190373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:59.289626 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:59.289589 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:59.598134 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:59.598106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:08:59.598320 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:59.598106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:08:59.598320 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:59.598232 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:08:59.598320 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:59.598106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:08:59.598320 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:59.598281 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:08:59.598539 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:08:59.598353 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:08:59.897239 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:08:59.897141 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:00.281368 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:00.281335 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:00.679669 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:00.679589 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:01.086226 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:01.086198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:01.145655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:01.145628 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:01.286527 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:01.286496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:01.597685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:01.597657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:01.597685 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:01.597672 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:01.597893 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:01.597666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:01.597893 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:01.597760 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:09:01.597893 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:01.597812 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:09:01.597992 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:01.597905 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:09:01.629088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:01.629064 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:01.733768 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:01.733733 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:02.502139 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:02.502106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:02.701133 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:02.701096 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:02.801765 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:02.801685 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:03.089142 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:03.089062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:03.316253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:03.316229 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:03.598355 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:03.598326 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:03.598524 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:03.598326 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:03.598524 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:03.598419 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:09:03.598524 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:03.598325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:03.598524 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:03.598510 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:09:03.598677 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:03.598568 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:09:03.685596 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:03.685570 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:04.306583 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:04.306552 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:04.884733 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:04.884702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:04.985282 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:04.985256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.047513 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.047488 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.110370 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.110344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.184067 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.184010 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.296641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.296617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.396675 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.396645 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.567641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.567614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.598095 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.598078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:05.598210 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.598181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:05.598210 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:05.598183 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:09:05.598327 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:05.598285 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:09:05.598327 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.598317 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:05.598427 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:05.598390 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:09:05.633573 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.633555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.710739 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.710716 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.781573 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.781549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.890603 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.890539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.990714 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:05.990685 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:06.091288 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:06.091234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:06.580226 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:06.580190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:06.776694 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:06.776664 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:06.832887 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:06.832807 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:07.221065 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:07.220986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:07.589009 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:07.588981 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:07.597915 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:07.597894 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:07.598041 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:07.597900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:07.598041 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:07.597985 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:09:07.598128 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:07.597900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:07.598128 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:07.598047 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:09:07.598128 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:07.598119 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:09:07.652377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:07.652351 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:09.598290 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:09.598264 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:09.598290 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:09.598290 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:09.598678 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:09.598362 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:09:09.598678 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:09.598393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:09.598678 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:09.598459 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:09:09.598678 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:09.598542 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:09:09.819427 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:09.819382 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:09.921943 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:09.921850 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:11.597749 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:11.597712 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:11.597749 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:11.597753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:11.598171 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:11.597758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:11.598171 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:11.597839 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:09:11.598171 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:11.597949 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:09:11.598171 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:11.598001 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:09:13.598036 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:13.598008 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:13.598036 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:13.598036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:13.598437 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:13.598100 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:09:13.598437 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:13.598181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:13.598437 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:13.598263 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:09:13.598437 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:13.598365 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:09:15.091533 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:15.091502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:15.177091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:15.177050 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:15.272693 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:15.272664 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:15.373534 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:15.373459 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:15.597931 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:15.597898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:15.597931 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:15.597928 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:15.598174 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:15.597898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:15.598174 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:15.598016 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5m825" podUID="80d3ae4b-a4d1-4f35-8815-61d9a05251ba" Jul 07 19:09:15.598174 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:15.598144 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jp8h" podUID="969a6be0-5242-4a03-9c44-befc4165567b" Jul 07 19:09:15.598333 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:15.598208 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wv6r9" podUID="9c4370b6-71fb-4592-8bf4-f9060e2448c5" Jul 07 19:09:15.936248 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:15.936215 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:16.036496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.036462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:16.261937 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.261907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:16.352140 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.352113 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-59.ec2.internal" event="NodeReady" Jul 07 19:09:16.352298 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.352231 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 19:09:16.364864 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.364838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:16.421747 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.421695 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j6rmz" podStartSLOduration=30.343727219 podStartE2EDuration="1m1.421679328s" totalImagesPullingTime="31.077952109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:53.969935833 +0000 UTC m=+39.905561194" watchObservedRunningTime="2026-07-07 19:09:16.421679328 +0000 UTC m=+62.357304688" Jul 07 19:09:16.422290 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.422275 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kgnws"] Jul 07 19:09:16.424331 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.424317 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.425926 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.425906 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rl9h8"] Jul 07 19:09:16.427563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.427530 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 19:09:16.427563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.427545 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 19:09:16.427707 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.427566 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kjmsg\"" Jul 07 19:09:16.427707 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.427619 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 19:09:16.427707 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.427698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rl9h8" Jul 07 19:09:16.427830 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.427814 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 19:09:16.430519 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.430490 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 19:09:16.430519 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.430499 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jbcxp\"" Jul 07 19:09:16.430519 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.430514 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 19:09:16.431150 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.431121 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 19:09:16.438809 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.438788 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kgnws"] Jul 07 19:09:16.440705 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.440686 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rl9h8"] Jul 07 19:09:16.470071 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.470044 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:16.485303 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.485282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deda2e77-de33-44a9-840e-bdc31a761927-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.485383 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.485308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqg4m\" (UniqueName: \"kubernetes.io/projected/deda2e77-de33-44a9-840e-bdc31a761927-kube-api-access-cqg4m\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.485383 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.485326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1559affd-634a-494c-935c-8e553c347f86-cert\") pod \"ingress-canary-rl9h8\" (UID: \"1559affd-634a-494c-935c-8e553c347f86\") " pod="openshift-ingress-canary/ingress-canary-rl9h8" Jul 07 19:09:16.485473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.485412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/deda2e77-de33-44a9-840e-bdc31a761927-crio-socket\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.485473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.485427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/deda2e77-de33-44a9-840e-bdc31a761927-data-volume\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.485473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.485444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/deda2e77-de33-44a9-840e-bdc31a761927-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.485473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.485462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64j9f\" (UniqueName: \"kubernetes.io/projected/1559affd-634a-494c-935c-8e553c347f86-kube-api-access-64j9f\") pod \"ingress-canary-rl9h8\" (UID: \"1559affd-634a-494c-935c-8e553c347f86\") " pod="openshift-ingress-canary/ingress-canary-rl9h8" Jul 07 19:09:16.518724 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.518655 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w94kr"] Jul 07 19:09:16.520474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.520460 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.523696 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.523679 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42k7f\"" Jul 07 19:09:16.523797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.523709 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 19:09:16.523797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.523730 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 19:09:16.532371 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.532353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w94kr"] Jul 07 19:09:16.563558 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.563502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:16.586200 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1559affd-634a-494c-935c-8e553c347f86-cert\") pod \"ingress-canary-rl9h8\" (UID: \"1559affd-634a-494c-935c-8e553c347f86\") " pod="openshift-ingress-canary/ingress-canary-rl9h8" Jul 07 19:09:16.586296 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4dr\" (UniqueName: \"kubernetes.io/projected/819038a7-6012-4c2f-964a-6f2c0e8b3359-kube-api-access-hc4dr\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.586296 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/819038a7-6012-4c2f-964a-6f2c0e8b3359-metrics-tls\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.586387 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819038a7-6012-4c2f-964a-6f2c0e8b3359-tmp-dir\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.586387 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/deda2e77-de33-44a9-840e-bdc31a761927-crio-socket\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.586387 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/deda2e77-de33-44a9-840e-bdc31a761927-data-volume\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.586387 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/deda2e77-de33-44a9-840e-bdc31a761927-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.586572 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64j9f\" (UniqueName: \"kubernetes.io/projected/1559affd-634a-494c-935c-8e553c347f86-kube-api-access-64j9f\") pod \"ingress-canary-rl9h8\" (UID: \"1559affd-634a-494c-935c-8e553c347f86\") " pod="openshift-ingress-canary/ingress-canary-rl9h8" Jul 07 19:09:16.586572 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/deda2e77-de33-44a9-840e-bdc31a761927-crio-socket\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.586572 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/819038a7-6012-4c2f-964a-6f2c0e8b3359-config-volume\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.586572 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deda2e77-de33-44a9-840e-bdc31a761927-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.586572 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqg4m\" (UniqueName: \"kubernetes.io/projected/deda2e77-de33-44a9-840e-bdc31a761927-kube-api-access-cqg4m\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.586816 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586590 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/deda2e77-de33-44a9-840e-bdc31a761927-data-volume\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.586975 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.586959 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/deda2e77-de33-44a9-840e-bdc31a761927-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.590263 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.590247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deda2e77-de33-44a9-840e-bdc31a761927-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.590314 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.590304 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1559affd-634a-494c-935c-8e553c347f86-cert\") pod \"ingress-canary-rl9h8\" (UID: \"1559affd-634a-494c-935c-8e553c347f86\") " pod="openshift-ingress-canary/ingress-canary-rl9h8" Jul 07 19:09:16.595370 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.595351 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64j9f\" (UniqueName: \"kubernetes.io/projected/1559affd-634a-494c-935c-8e553c347f86-kube-api-access-64j9f\") pod \"ingress-canary-rl9h8\" (UID: \"1559affd-634a-494c-935c-8e553c347f86\") " pod="openshift-ingress-canary/ingress-canary-rl9h8" Jul 07 19:09:16.595776 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.595757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqg4m\" (UniqueName: \"kubernetes.io/projected/deda2e77-de33-44a9-840e-bdc31a761927-kube-api-access-cqg4m\") pod \"insights-runtime-extractor-kgnws\" (UID: \"deda2e77-de33-44a9-840e-bdc31a761927\") " pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.650196 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.650171 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:16.687421 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.687383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4dr\" (UniqueName: \"kubernetes.io/projected/819038a7-6012-4c2f-964a-6f2c0e8b3359-kube-api-access-hc4dr\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.687531 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.687469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/819038a7-6012-4c2f-964a-6f2c0e8b3359-metrics-tls\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.687531 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.687497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819038a7-6012-4c2f-964a-6f2c0e8b3359-tmp-dir\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.687641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.687532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/819038a7-6012-4c2f-964a-6f2c0e8b3359-config-volume\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.687824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.687806 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819038a7-6012-4c2f-964a-6f2c0e8b3359-tmp-dir\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.688110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.688092 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/819038a7-6012-4c2f-964a-6f2c0e8b3359-config-volume\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.689797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.689778 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/819038a7-6012-4c2f-964a-6f2c0e8b3359-metrics-tls\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.695419 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.695397 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4dr\" (UniqueName: \"kubernetes.io/projected/819038a7-6012-4c2f-964a-6f2c0e8b3359-kube-api-access-hc4dr\") pod \"dns-default-w94kr\" (UID: \"819038a7-6012-4c2f-964a-6f2c0e8b3359\") " pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.721187 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.721156 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:16.734128 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.734111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kgnws" Jul 07 19:09:16.738508 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.738495 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rl9h8" Jul 07 19:09:16.829304 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.829187 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:16.885117 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.885085 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kgnws"] Jul 07 19:09:16.912855 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.912804 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rl9h8"] Jul 07 19:09:16.915037 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:16.915014 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1559affd_634a_494c_935c_8e553c347f86.slice/crio-0b12a65db01a6b3b876bc7da04c2cfd321a14a26e3dc70d7c7e4f4b366460c15 WatchSource:0}: Error finding container 0b12a65db01a6b3b876bc7da04c2cfd321a14a26e3dc70d7c7e4f4b366460c15: Status 404 returned error can't find the container with id 0b12a65db01a6b3b876bc7da04c2cfd321a14a26e3dc70d7c7e4f4b366460c15 Jul 07 19:09:16.974474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.974447 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w94kr"] Jul 07 19:09:16.977015 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:16.976990 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819038a7_6012_4c2f_964a_6f2c0e8b3359.slice/crio-886198574f0093866fea30621ef7f90d2e3e53902209448aa2db8c3b95e99a71 WatchSource:0}: Error finding container 886198574f0093866fea30621ef7f90d2e3e53902209448aa2db8c3b95e99a71: Status 404 returned error can't find the container with id 886198574f0093866fea30621ef7f90d2e3e53902209448aa2db8c3b95e99a71 Jul 07 19:09:16.981143 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.981115 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:16.993626 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.993605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w94kr" event={"ID":"819038a7-6012-4c2f-964a-6f2c0e8b3359","Type":"ContainerStarted","Data":"886198574f0093866fea30621ef7f90d2e3e53902209448aa2db8c3b95e99a71"} Jul 07 19:09:16.994649 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.994630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rl9h8" event={"ID":"1559affd-634a-494c-935c-8e553c347f86","Type":"ContainerStarted","Data":"0b12a65db01a6b3b876bc7da04c2cfd321a14a26e3dc70d7c7e4f4b366460c15"} Jul 07 19:09:16.995684 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:16.995664 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgnws" event={"ID":"deda2e77-de33-44a9-840e-bdc31a761927","Type":"ContainerStarted","Data":"f3b49e872c88a0c0c0c25617f9e2398f24f851dabaafbef780dd5678cdef02a6"} Jul 07 19:09:17.600984 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:17.597692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:17.600984 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:17.598138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:17.600984 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:17.598454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:17.601798 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:17.601777 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 19:09:17.603086 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:17.603063 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 19:09:17.603319 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:17.603302 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7q8zk\"" Jul 07 19:09:17.603614 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:17.603598 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 19:09:17.603909 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:17.603891 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xjkd\"" Jul 07 19:09:17.604134 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:17.604120 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 19:09:18.046812 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.046082 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d6dff4898-vncp9"] Jul 07 19:09:18.049192 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.048192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.052372 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.052347 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 19:09:18.053512 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.053489 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 19:09:18.053832 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.053816 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 19:09:18.054825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.054053 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 19:09:18.054825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.054323 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j4wl4\"" Jul 07 19:09:18.054825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.054487 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 19:09:18.054825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.054498 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 19:09:18.054825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.054690 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 19:09:18.064057 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.064038 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 19:09:18.067323 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.067305 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6dff4898-vncp9"] Jul 07 19:09:18.068952 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.068897 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:18.109946 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.109894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-oauth-config\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.109946 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.109934 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-trusted-ca-bundle\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.110151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.109985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-serving-cert\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.110151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.110009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-service-ca\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.110151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.110039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsk2j\" (UniqueName: \"kubernetes.io/projected/66d33a38-5a79-4b49-ba45-a06d28688968-kube-api-access-zsk2j\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.110151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.110092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-console-config\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.110151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.110132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-oauth-serving-cert\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.154648 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.154569 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:18.210678 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.210645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-console-config\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.210864 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.210705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-oauth-serving-cert\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.210864 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.210744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-oauth-config\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.210864 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.210768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-trusted-ca-bundle\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.210864 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.210812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-serving-cert\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.211121 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.210905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-service-ca\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.211121 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.210935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsk2j\" (UniqueName: \"kubernetes.io/projected/66d33a38-5a79-4b49-ba45-a06d28688968-kube-api-access-zsk2j\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.211546 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.211497 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-console-config\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.211648 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.211544 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-oauth-serving-cert\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.211714 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.211649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-trusted-ca-bundle\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.212431 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.212401 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-service-ca\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.214424 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.214381 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-oauth-config\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.221515 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.221488 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-serving-cert\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.226035 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.226011 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsk2j\" (UniqueName: \"kubernetes.io/projected/66d33a38-5a79-4b49-ba45-a06d28688968-kube-api-access-zsk2j\") pod \"console-7d6dff4898-vncp9\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.363778 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.363695 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:18.526156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.526123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6dff4898-vncp9"] Jul 07 19:09:18.938334 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:18.938304 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhfhb" Jul 07 19:09:19.183245 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:19.183213 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:19.371026 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:19.370998 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:19.456362 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:19.456326 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:19.882571 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:19.882534 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:19.982637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:19.982592 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:20.229639 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:20.229564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:20.231847 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:20.231827 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969a6be0-5242-4a03-9c44-befc4165567b-metrics-certs\") pod \"network-metrics-daemon-9jp8h\" (UID: \"969a6be0-5242-4a03-9c44-befc4165567b\") " pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:20.246455 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:20.246430 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d33a38_5a79_4b49_ba45_a06d28688968.slice/crio-576a278264a173584270dfc2175452ac863eeb3750f10c218468784ece168d99 WatchSource:0}: Error finding container 576a278264a173584270dfc2175452ac863eeb3750f10c218468784ece168d99: Status 404 returned error can't find the container with id 576a278264a173584270dfc2175452ac863eeb3750f10c218468784ece168d99 Jul 07 19:09:20.329955 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:20.329920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:20.329955 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:20.329960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:20.332553 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:20.332527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80d3ae4b-a4d1-4f35-8815-61d9a05251ba-original-pull-secret\") pod \"global-pull-secret-syncer-5m825\" (UID: \"80d3ae4b-a4d1-4f35-8815-61d9a05251ba\") " pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:20.332834 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:20.332812 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6tl\" (UniqueName: \"kubernetes.io/projected/9c4370b6-71fb-4592-8bf4-f9060e2448c5-kube-api-access-xg6tl\") pod \"network-check-target-wv6r9\" (UID: \"9c4370b6-71fb-4592-8bf4-f9060e2448c5\") " pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:20.334490 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:20.334471 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:20.341566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:20.341542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jp8h" Jul 07 19:09:20.626832 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:20.626799 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5m825" Jul 07 19:09:21.009176 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:21.009135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6dff4898-vncp9" event={"ID":"66d33a38-5a79-4b49-ba45-a06d28688968","Type":"ContainerStarted","Data":"576a278264a173584270dfc2175452ac863eeb3750f10c218468784ece168d99"} Jul 07 19:09:22.202584 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:22.202495 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:22.269616 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:22.269584 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:22.484075 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:22.483960 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5m825"] Jul 07 19:09:22.498905 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:22.498862 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wv6r9"] Jul 07 19:09:22.704358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:22.704330 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9jp8h"] Jul 07 19:09:23.017392 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.017352 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w94kr" event={"ID":"819038a7-6012-4c2f-964a-6f2c0e8b3359","Type":"ContainerStarted","Data":"a2219bd14d5e021368bf5dd29ace96011638e39f4bd153afbf2dcc5c4057d158"} Jul 07 19:09:23.217016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.216979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:23.291303 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.290413 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6"] Jul 07 19:09:23.296718 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.296682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.300560 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.300532 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 19:09:23.300965 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.300942 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 19:09:23.301080 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.300973 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 19:09:23.301080 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.301023 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 19:09:23.301193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.301095 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nwfjq\"" Jul 07 19:09:23.301193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.301181 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 19:09:23.305604 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.305584 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6"] Jul 07 19:09:23.313273 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.313241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:23.350064 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.350030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1119552f-394b-40bb-b13f-8c3235e908f1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.350217 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.350081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1119552f-394b-40bb-b13f-8c3235e908f1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.350217 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.350186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1119552f-394b-40bb-b13f-8c3235e908f1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.350217 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.350210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwh89\" (UniqueName: \"kubernetes.io/projected/1119552f-394b-40bb-b13f-8c3235e908f1-kube-api-access-zwh89\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.450688 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.450645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1119552f-394b-40bb-b13f-8c3235e908f1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.450836 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.450697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwh89\" (UniqueName: \"kubernetes.io/projected/1119552f-394b-40bb-b13f-8c3235e908f1-kube-api-access-zwh89\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.450836 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.450724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1119552f-394b-40bb-b13f-8c3235e908f1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.450836 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.450745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1119552f-394b-40bb-b13f-8c3235e908f1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.451459 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.451434 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1119552f-394b-40bb-b13f-8c3235e908f1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.451708 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:23.451673 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 07 19:09:23.451826 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:23.451759 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1119552f-394b-40bb-b13f-8c3235e908f1-prometheus-operator-tls podName:1119552f-394b-40bb-b13f-8c3235e908f1 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:23.95173911 +0000 UTC m=+69.887364461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1119552f-394b-40bb-b13f-8c3235e908f1-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-vnsh6" (UID: "1119552f-394b-40bb-b13f-8c3235e908f1") : secret "prometheus-operator-tls" not found Jul 07 19:09:23.453973 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.453952 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1119552f-394b-40bb-b13f-8c3235e908f1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.462374 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.462348 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwh89\" (UniqueName: \"kubernetes.io/projected/1119552f-394b-40bb-b13f-8c3235e908f1-kube-api-access-zwh89\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.495165 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:23.495134 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969a6be0_5242_4a03_9c44_befc4165567b.slice/crio-72b035a0d392387944697185962ea283a3817b5f7cfdffb1f0ce4fdd38a399d1 WatchSource:0}: Error finding container 72b035a0d392387944697185962ea283a3817b5f7cfdffb1f0ce4fdd38a399d1: Status 404 returned error can't find the container with id 72b035a0d392387944697185962ea283a3817b5f7cfdffb1f0ce4fdd38a399d1 Jul 07 19:09:23.495839 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:23.495811 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d3ae4b_a4d1_4f35_8815_61d9a05251ba.slice/crio-3adaf61632b3b299feeb97d6054d08feaeb82e4b98d41e871eea5503d180dc0e WatchSource:0}: Error finding container 3adaf61632b3b299feeb97d6054d08feaeb82e4b98d41e871eea5503d180dc0e: Status 404 returned error can't find the container with id 3adaf61632b3b299feeb97d6054d08feaeb82e4b98d41e871eea5503d180dc0e Jul 07 19:09:23.496714 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:23.496674 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4370b6_71fb_4592_8bf4_f9060e2448c5.slice/crio-65e7ecc91962c865d7c19bc07be00f15d875d02c7d5f1406776b84fff30e36c2 WatchSource:0}: Error finding container 65e7ecc91962c865d7c19bc07be00f15d875d02c7d5f1406776b84fff30e36c2: Status 404 returned error can't find the container with id 65e7ecc91962c865d7c19bc07be00f15d875d02c7d5f1406776b84fff30e36c2 Jul 07 19:09:23.561553 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.561529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:23.954522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.954436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1119552f-394b-40bb-b13f-8c3235e908f1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:23.956821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:23.956802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1119552f-394b-40bb-b13f-8c3235e908f1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vnsh6\" (UID: \"1119552f-394b-40bb-b13f-8c3235e908f1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:24.021350 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.021306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w94kr" event={"ID":"819038a7-6012-4c2f-964a-6f2c0e8b3359","Type":"ContainerStarted","Data":"77c9b0ee9bd55cdf8f65cede81e9b53c80e2f95340857d3eac36745bc97a7b1b"} Jul 07 19:09:24.021524 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.021430 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:24.022729 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.022703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6dff4898-vncp9" event={"ID":"66d33a38-5a79-4b49-ba45-a06d28688968","Type":"ContainerStarted","Data":"154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935"} Jul 07 19:09:24.023813 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.023788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5m825" event={"ID":"80d3ae4b-a4d1-4f35-8815-61d9a05251ba","Type":"ContainerStarted","Data":"3adaf61632b3b299feeb97d6054d08feaeb82e4b98d41e871eea5503d180dc0e"} Jul 07 19:09:24.025136 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.025117 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgnws" event={"ID":"deda2e77-de33-44a9-840e-bdc31a761927","Type":"ContainerStarted","Data":"1cc273a24d4cdd79ea862ce8dee58adaaea8725892d6cac3221323a83b33582e"} Jul 07 19:09:24.026219 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.026202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wv6r9" event={"ID":"9c4370b6-71fb-4592-8bf4-f9060e2448c5","Type":"ContainerStarted","Data":"65e7ecc91962c865d7c19bc07be00f15d875d02c7d5f1406776b84fff30e36c2"} Jul 07 19:09:24.027192 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.027175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9jp8h" event={"ID":"969a6be0-5242-4a03-9c44-befc4165567b","Type":"ContainerStarted","Data":"72b035a0d392387944697185962ea283a3817b5f7cfdffb1f0ce4fdd38a399d1"} Jul 07 19:09:24.028447 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.028430 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rl9h8" event={"ID":"1559affd-634a-494c-935c-8e553c347f86","Type":"ContainerStarted","Data":"c3b95eb6c0efc7a54a944252e3bcf6e0d913c404a01f57dce4d1dfe29b9448a3"} Jul 07 19:09:24.041719 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.041676 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w94kr" podStartSLOduration=3.17391915 podStartE2EDuration="8.041661604s" totalImagesPullingTime="4.867742454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:24.041003 +0000 UTC m=+69.976628361" watchObservedRunningTime="2026-07-07 19:09:24.041661604 +0000 UTC m=+69.977286958" Jul 07 19:09:24.061662 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.061613 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d6dff4898-vncp9" podStartSLOduration=2.798471701 podStartE2EDuration="6.061597419s" totalImagesPullingTime="3.263125718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:24.060534225 +0000 UTC m=+69.996159585" watchObservedRunningTime="2026-07-07 19:09:24.061597419 +0000 UTC m=+69.997222759" Jul 07 19:09:24.082131 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.081778 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rl9h8" podStartSLOduration=2.667900259 podStartE2EDuration="8.081757018s" totalImagesPullingTime="5.413856759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:24.07730635 +0000 UTC m=+70.012931711" watchObservedRunningTime="2026-07-07 19:09:24.081757018 +0000 UTC m=+70.017382362" Jul 07 19:09:24.207408 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.207323 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" Jul 07 19:09:24.445352 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:24.445319 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6"] Jul 07 19:09:24.449388 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:24.449358 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1119552f_394b_40bb_b13f_8c3235e908f1.slice/crio-548dbefa804eb1f01927d45cd5327101260c3b7e99089bf22fd764a02d2ae318 WatchSource:0}: Error finding container 548dbefa804eb1f01927d45cd5327101260c3b7e99089bf22fd764a02d2ae318: Status 404 returned error can't find the container with id 548dbefa804eb1f01927d45cd5327101260c3b7e99089bf22fd764a02d2ae318 Jul 07 19:09:25.039521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:25.039379 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" event={"ID":"1119552f-394b-40bb-b13f-8c3235e908f1","Type":"ContainerStarted","Data":"548dbefa804eb1f01927d45cd5327101260c3b7e99089bf22fd764a02d2ae318"} Jul 07 19:09:25.042144 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:25.042116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgnws" event={"ID":"deda2e77-de33-44a9-840e-bdc31a761927","Type":"ContainerStarted","Data":"f6a64661c058965c19e32b5922633815adcdd93f95d7a0767dacb84b911a9b46"} Jul 07 19:09:25.046490 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:25.046368 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9jp8h" event={"ID":"969a6be0-5242-4a03-9c44-befc4165567b","Type":"ContainerStarted","Data":"b196f2eaa6ac2f64eb3a4aab5ff9840f43aacbe8ac0eff1a47dcd0f72dd2ea46"} Jul 07 19:09:25.046490 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:25.046412 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9jp8h" event={"ID":"969a6be0-5242-4a03-9c44-befc4165567b","Type":"ContainerStarted","Data":"b35bace92512428b9133e797b2eb49a66f2571af5232829284eb19e1ac496e7a"} Jul 07 19:09:25.067117 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:25.067069 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9jp8h" podStartSLOduration=68.796504977 podStartE2EDuration="1m10.06705142s" totalImagesPullingTime="1.270546443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:25.065638289 +0000 UTC m=+71.001263649" watchObservedRunningTime="2026-07-07 19:09:25.06705142 +0000 UTC m=+71.002676781" Jul 07 19:09:25.164823 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:25.164780 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:25.267166 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:25.267137 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:25.420583 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:25.420505 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:25.493990 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:25.493801 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:28.364739 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:28.364685 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:28.365259 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:28.364758 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:28.366427 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:28.366387 2577 patch_prober.go:28] interesting pod/console-7d6dff4898-vncp9 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 07 19:09:28.366561 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:28.366445 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7d6dff4898-vncp9" podUID="66d33a38-5a79-4b49-ba45-a06d28688968" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 07 19:09:30.067093 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.067050 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgnws" event={"ID":"deda2e77-de33-44a9-840e-bdc31a761927","Type":"ContainerStarted","Data":"17809a48f54d3b133c806ba711f60b72ef0234f767a8bdd7ab90f2bd510e015d"} Jul 07 19:09:30.068371 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.068326 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wv6r9" event={"ID":"9c4370b6-71fb-4592-8bf4-f9060e2448c5","Type":"ContainerStarted","Data":"8564fe6e8d8644fdfc5581a46c7a467795e48b5f6339338a93b92dfed51f1db4"} Jul 07 19:09:30.068496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.068449 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:09:30.069857 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.069806 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5m825" event={"ID":"80d3ae4b-a4d1-4f35-8815-61d9a05251ba","Type":"ContainerStarted","Data":"03bd741c51ee7572454a22e8f9a4fc611940fb20acc24dfbc855f39119a80335"} Jul 07 19:09:30.072119 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.072021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" event={"ID":"1119552f-394b-40bb-b13f-8c3235e908f1","Type":"ContainerStarted","Data":"8a3e9b1574b2e8c3f9ab8e74ad246f2b0cf752335d3d121e74c0e0b614591e51"} Jul 07 19:09:30.072119 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.072063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" event={"ID":"1119552f-394b-40bb-b13f-8c3235e908f1","Type":"ContainerStarted","Data":"d8e3b2155b98a919b0a81615a29117306da20225b454c2634aa4eed281c9343f"} Jul 07 19:09:30.087919 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.087842 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kgnws" podStartSLOduration=1.296552063 podStartE2EDuration="14.087825695s" totalImagesPullingTime="12.791273632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:30.086523743 +0000 UTC m=+76.022149103" watchObservedRunningTime="2026-07-07 19:09:30.087825695 +0000 UTC m=+76.023451056" Jul 07 19:09:30.106891 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.106784 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vnsh6" podStartSLOduration=1.689683383 podStartE2EDuration="7.106766505s" totalImagesPullingTime="5.417083122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:30.105478127 +0000 UTC m=+76.041103477" watchObservedRunningTime="2026-07-07 19:09:30.106766505 +0000 UTC m=+76.042391867" Jul 07 19:09:30.127136 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.127089 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wv6r9" podStartSLOduration=69.757970445 podStartE2EDuration="1m16.127075789s" totalImagesPullingTime="6.369105344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:30.125586955 +0000 UTC m=+76.061212314" watchObservedRunningTime="2026-07-07 19:09:30.127075789 +0000 UTC m=+76.062701148" Jul 07 19:09:30.145115 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.145055 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5m825" podStartSLOduration=68.768977212 podStartE2EDuration="1m15.145038509s" totalImagesPullingTime="6.376061297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:30.144929965 +0000 UTC m=+76.080555325" watchObservedRunningTime="2026-07-07 19:09:30.145038509 +0000 UTC m=+76.080663869" Jul 07 19:09:30.236794 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:30.236739 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:31.393517 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.393484 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:31.589739 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.589708 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:31.667079 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.667002 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg"] Jul 07 19:09:31.670251 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.670235 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.672788 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.672768 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 19:09:31.673200 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.673183 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hzb69\"" Jul 07 19:09:31.673355 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.673342 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 19:09:31.681938 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.681918 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xwngj"] Jul 07 19:09:31.684934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.684916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.686827 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.686809 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg"] Jul 07 19:09:31.687532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.687511 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 19:09:31.687697 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.687674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 19:09:31.687956 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.687940 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7h7cz\"" Jul 07 19:09:31.688047 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.687956 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 19:09:31.709736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.709708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-accelerators-collector-config\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.709866 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.709740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-tls\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.709866 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.709758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/830d4717-38fd-4379-ac29-bb324132289b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.709866 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.709778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf7gm\" (UniqueName: \"kubernetes.io/projected/830d4717-38fd-4379-ac29-bb324132289b-kube-api-access-nf7gm\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.709866 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.709852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-wtmp\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.710087 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.709907 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/703f6de1-91b9-4fd0-8f74-c617e8968f09-root\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.710087 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.709929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.710087 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.709976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-textfile\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.710087 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.709991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brp2t\" (UniqueName: \"kubernetes.io/projected/703f6de1-91b9-4fd0-8f74-c617e8968f09-kube-api-access-brp2t\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.710087 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.710038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/703f6de1-91b9-4fd0-8f74-c617e8968f09-sys\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.710280 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.710087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/830d4717-38fd-4379-ac29-bb324132289b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.710280 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.710117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/830d4717-38fd-4379-ac29-bb324132289b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.710280 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.710152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/703f6de1-91b9-4fd0-8f74-c617e8968f09-metrics-client-ca\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.712401 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.712381 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6"] Jul 07 19:09:31.715526 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.715512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.718430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.718411 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 19:09:31.718539 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.718519 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zc7sq\"" Jul 07 19:09:31.718636 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.718547 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 19:09:31.718801 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.718784 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 19:09:31.729286 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.729253 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:31.729497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.729482 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6"] Jul 07 19:09:31.810827 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.810792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/703f6de1-91b9-4fd0-8f74-c617e8968f09-metrics-client-ca\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.810841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-accelerators-collector-config\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.810860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-tls\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.810896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/830d4717-38fd-4379-ac29-bb324132289b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.811045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.810916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf7gm\" (UniqueName: \"kubernetes.io/projected/830d4717-38fd-4379-ac29-bb324132289b-kube-api-access-nf7gm\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.811045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.810935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-wtmp\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.810956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rr9s\" (UniqueName: \"kubernetes.io/projected/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-api-access-5rr9s\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811072 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69501978-25c4-4e80-bb45-009fdc2e3a84-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/703f6de1-91b9-4fd0-8f74-c617e8968f09-root\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811117 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-wtmp\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-textfile\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brp2t\" (UniqueName: \"kubernetes.io/projected/703f6de1-91b9-4fd0-8f74-c617e8968f09-kube-api-access-brp2t\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/703f6de1-91b9-4fd0-8f74-c617e8968f09-sys\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.811340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/830d4717-38fd-4379-ac29-bb324132289b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.812101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/830d4717-38fd-4379-ac29-bb324132289b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.812101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/69501978-25c4-4e80-bb45-009fdc2e3a84-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.812101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811504 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/703f6de1-91b9-4fd0-8f74-c617e8968f09-metrics-client-ca\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.812101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811504 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-accelerators-collector-config\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.812101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811598 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/703f6de1-91b9-4fd0-8f74-c617e8968f09-sys\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.812101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811666 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-textfile\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.812101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811699 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/830d4717-38fd-4379-ac29-bb324132289b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.812101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.811948 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/703f6de1-91b9-4fd0-8f74-c617e8968f09-root\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.813568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.813542 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-tls\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.813668 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.813635 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/703f6de1-91b9-4fd0-8f74-c617e8968f09-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.813779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.813761 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/830d4717-38fd-4379-ac29-bb324132289b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.814100 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.814080 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/830d4717-38fd-4379-ac29-bb324132289b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.830546 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.830523 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf7gm\" (UniqueName: \"kubernetes.io/projected/830d4717-38fd-4379-ac29-bb324132289b-kube-api-access-nf7gm\") pod \"openshift-state-metrics-54cf96dfcf-kzplg\" (UID: \"830d4717-38fd-4379-ac29-bb324132289b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.830996 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.830971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brp2t\" (UniqueName: \"kubernetes.io/projected/703f6de1-91b9-4fd0-8f74-c617e8968f09-kube-api-access-brp2t\") pod \"node-exporter-xwngj\" (UID: \"703f6de1-91b9-4fd0-8f74-c617e8968f09\") " pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:31.833782 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.833760 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:31.912065 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.912027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rr9s\" (UniqueName: \"kubernetes.io/projected/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-api-access-5rr9s\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.912065 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.912064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.912312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.912082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69501978-25c4-4e80-bb45-009fdc2e3a84-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.912312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.912112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.912312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.912130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.912312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.912164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/69501978-25c4-4e80-bb45-009fdc2e3a84-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.912595 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.912578 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/69501978-25c4-4e80-bb45-009fdc2e3a84-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.912841 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.912810 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.912971 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.912847 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69501978-25c4-4e80-bb45-009fdc2e3a84-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.914852 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.914828 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.914852 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.914845 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.925541 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.925480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rr9s\" (UniqueName: \"kubernetes.io/projected/69501978-25c4-4e80-bb45-009fdc2e3a84-kube-api-access-5rr9s\") pod \"kube-state-metrics-7c9dbb8b7d-mk2p6\" (UID: \"69501978-25c4-4e80-bb45-009fdc2e3a84\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:31.931867 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.931826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:31.978352 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.978311 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" Jul 07 19:09:31.995496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:31.995469 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xwngj" Jul 07 19:09:32.006743 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:32.006717 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703f6de1_91b9_4fd0_8f74_c617e8968f09.slice/crio-f4c31d960eb89c27707cd67d83d24dc145410afef4698ef66cdbe8aef2e72381 WatchSource:0}: Error finding container f4c31d960eb89c27707cd67d83d24dc145410afef4698ef66cdbe8aef2e72381: Status 404 returned error can't find the container with id f4c31d960eb89c27707cd67d83d24dc145410afef4698ef66cdbe8aef2e72381 Jul 07 19:09:32.024099 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.024070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" Jul 07 19:09:32.032314 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.032284 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:32.080544 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.079797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xwngj" event={"ID":"703f6de1-91b9-4fd0-8f74-c617e8968f09","Type":"ContainerStarted","Data":"f4c31d960eb89c27707cd67d83d24dc145410afef4698ef66cdbe8aef2e72381"} Jul 07 19:09:32.137211 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.137179 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg"] Jul 07 19:09:32.139775 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:32.139751 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830d4717_38fd_4379_ac29_bb324132289b.slice/crio-5f8ce2196b2ec2335cb2da83d94ee16d426d65ebe8c5f9adb174a9500f3e4cdf WatchSource:0}: Error finding container 5f8ce2196b2ec2335cb2da83d94ee16d426d65ebe8c5f9adb174a9500f3e4cdf: Status 404 returned error can't find the container with id 5f8ce2196b2ec2335cb2da83d94ee16d426d65ebe8c5f9adb174a9500f3e4cdf Jul 07 19:09:32.178155 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.178098 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6"] Jul 07 19:09:32.180650 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:32.180628 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69501978_25c4_4e80_bb45_009fdc2e3a84.slice/crio-fb44f939b79a8310ea474a7cb41bd2ed88ac51ce6d3394da83e5d4762ae73b43 WatchSource:0}: Error finding container fb44f939b79a8310ea474a7cb41bd2ed88ac51ce6d3394da83e5d4762ae73b43: Status 404 returned error can't find the container with id fb44f939b79a8310ea474a7cb41bd2ed88ac51ce6d3394da83e5d4762ae73b43 Jul 07 19:09:32.184516 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.184458 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:32.696642 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.696606 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:32.797815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.797776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:32.823735 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.823703 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:09:32.830380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.829190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.834372 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.834344 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 19:09:32.834514 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.834435 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mqthr\"" Jul 07 19:09:32.834514 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.834442 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 19:09:32.834637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.834550 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 19:09:32.834637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.834604 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 19:09:32.834637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.834626 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 19:09:32.834775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.834684 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 19:09:32.834894 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.834842 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 19:09:32.834894 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.834843 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 19:09:32.839897 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.839858 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 19:09:32.852253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.852220 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:09:32.923417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zp5j\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-kube-api-access-2zp5j\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923607 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923607 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923607 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-web-config\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923607 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923550 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923607 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923607 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923944 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923944 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923944 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923944 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-config-out\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923944 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:32.923944 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:32.923790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024233 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-config-out\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zp5j\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-kube-api-access-2zp5j\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-web-config\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.024775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.024587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.025836 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:33.025646 2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 19:09:33.025836 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:33.025723 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-main-tls podName:0b4eef57-d363-4617-be33-6d0a5713bebb nodeName:}" failed. No retries permitted until 2026-07-07 19:09:33.525700802 +0000 UTC m=+79.461326143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb") : secret "alertmanager-main-tls" not found Jul 07 19:09:33.026394 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.026065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.026394 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:33.026084 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-trusted-ca-bundle podName:0b4eef57-d363-4617-be33-6d0a5713bebb nodeName:}" failed. No retries permitted until 2026-07-07 19:09:33.526068613 +0000 UTC m=+79.461693968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb") : configmap references non-existent config key: ca-bundle.crt Jul 07 19:09:33.027358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.027335 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.031565 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.031495 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.032046 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.032002 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.033754 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.033733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-web-config\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.036391 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.036367 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.036934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.036909 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.037480 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.037455 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.038236 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.038208 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.040716 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.040530 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-config-out\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.040716 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.040539 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zp5j\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-kube-api-access-2zp5j\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.085719 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.085681 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" event={"ID":"69501978-25c4-4e80-bb45-009fdc2e3a84","Type":"ContainerStarted","Data":"fb44f939b79a8310ea474a7cb41bd2ed88ac51ce6d3394da83e5d4762ae73b43"} Jul 07 19:09:33.088859 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.088826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" event={"ID":"830d4717-38fd-4379-ac29-bb324132289b","Type":"ContainerStarted","Data":"dd0395e0160515a66a7275de241c399251a312f330c1911db2499bd7a16322fe"} Jul 07 19:09:33.089011 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.088912 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" event={"ID":"830d4717-38fd-4379-ac29-bb324132289b","Type":"ContainerStarted","Data":"86a2df1049006ec3117afb08d5351fb3745198f5bba1b8f4e899d044dd2223f1"} Jul 07 19:09:33.089011 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.088943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" event={"ID":"830d4717-38fd-4379-ac29-bb324132289b","Type":"ContainerStarted","Data":"5f8ce2196b2ec2335cb2da83d94ee16d426d65ebe8c5f9adb174a9500f3e4cdf"} Jul 07 19:09:33.275652 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.275571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:33.379918 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.379869 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:33.475375 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.475339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:33.529058 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.528976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.529058 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.529036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.530039 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.530013 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.531548 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.531527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.716413 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.716375 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8"] Jul 07 19:09:33.720461 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.720440 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.723549 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.723524 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 19:09:33.724236 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.724217 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 19:09:33.724330 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.724223 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 19:09:33.729328 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.729308 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bl7dnqvq48usf\"" Jul 07 19:09:33.729422 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.729334 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 19:09:33.729496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.729481 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 19:09:33.729595 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.729514 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4wnrh\"" Jul 07 19:09:33.736049 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.736026 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8"] Jul 07 19:09:33.741742 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.741724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:09:33.831589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.831568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.831681 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.831600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.831681 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.831621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.831681 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.831643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-metrics-client-ca\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.831827 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.831757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.831827 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.831790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfvgb\" (UniqueName: \"kubernetes.io/projected/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-kube-api-access-hfvgb\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.831926 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.831833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-grpc-tls\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.831926 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.831888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-tls\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.932312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.932281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-tls\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.932465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.932337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.932465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.932378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.932465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.932406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.932465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.932439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-metrics-client-ca\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.932692 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.932504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.932692 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.932528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfvgb\" (UniqueName: \"kubernetes.io/projected/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-kube-api-access-hfvgb\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.932692 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.932556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-grpc-tls\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.934478 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.934117 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-metrics-client-ca\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.937918 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.937894 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.938004 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.937919 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.938077 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.937999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-grpc-tls\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.938441 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.938404 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.938969 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.938864 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-tls\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.939515 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.939491 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:33.946332 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:33.946309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfvgb\" (UniqueName: \"kubernetes.io/projected/ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe-kube-api-access-hfvgb\") pod \"thanos-querier-7bcbcfdd9b-n26x8\" (UID: \"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe\") " pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:34.024115 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.024082 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:09:34.026650 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:34.026623 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4eef57_d363_4617_be33_6d0a5713bebb.slice/crio-6cab5f5a6a9dc4b0d9cc90ede75ba0d564df40edcb5632e9fd6c86f97b613c32 WatchSource:0}: Error finding container 6cab5f5a6a9dc4b0d9cc90ede75ba0d564df40edcb5632e9fd6c86f97b613c32: Status 404 returned error can't find the container with id 6cab5f5a6a9dc4b0d9cc90ede75ba0d564df40edcb5632e9fd6c86f97b613c32 Jul 07 19:09:34.033710 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.033118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:34.051207 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.050133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w94kr" Jul 07 19:09:34.098009 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.097783 2577 generic.go:356] "Generic (PLEG): container finished" podID="703f6de1-91b9-4fd0-8f74-c617e8968f09" containerID="7ea160b844490c4a585b2dd3e8a3bae0ff956fe7a3542ae9c2d0371a014f4e99" exitCode=0 Jul 07 19:09:34.098009 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.097846 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xwngj" event={"ID":"703f6de1-91b9-4fd0-8f74-c617e8968f09","Type":"ContainerDied","Data":"7ea160b844490c4a585b2dd3e8a3bae0ff956fe7a3542ae9c2d0371a014f4e99"} Jul 07 19:09:34.100797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.100377 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" event={"ID":"69501978-25c4-4e80-bb45-009fdc2e3a84","Type":"ContainerStarted","Data":"f5c08776c44d3fb711e50309af14dd589297809f2dd335aa8e82c96df38f522e"} Jul 07 19:09:34.100797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.100433 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" event={"ID":"69501978-25c4-4e80-bb45-009fdc2e3a84","Type":"ContainerStarted","Data":"b4a5ab686c5967123ba7802c963d217e820c3ecf6ce0e5c92d5836aeb513f47f"} Jul 07 19:09:34.105133 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.103232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" event={"ID":"830d4717-38fd-4379-ac29-bb324132289b","Type":"ContainerStarted","Data":"363fa44b2138e3abc1cba9b1f777541df491adfd6519f06e526ff83243c024cb"} Jul 07 19:09:34.105210 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.105137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerStarted","Data":"6cab5f5a6a9dc4b0d9cc90ede75ba0d564df40edcb5632e9fd6c86f97b613c32"} Jul 07 19:09:34.160225 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.157828 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kzplg" podStartSLOduration=1.546288386 podStartE2EDuration="3.157804123s" totalImagesPullingTime="1.611515737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:34.157073473 +0000 UTC m=+80.092698836" watchObservedRunningTime="2026-07-07 19:09:34.157804123 +0000 UTC m=+80.093429484" Jul 07 19:09:34.249075 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.249047 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8"] Jul 07 19:09:34.255849 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:34.255822 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba312e2d_2c39_45d6_bcd6_e9017a1c9ffe.slice/crio-ea6a214db8273ed4a7180fb1da93f73fc0f8c636fe60b81df975d5fc15b8c94f WatchSource:0}: Error finding container ea6a214db8273ed4a7180fb1da93f73fc0f8c636fe60b81df975d5fc15b8c94f: Status 404 returned error can't find the container with id ea6a214db8273ed4a7180fb1da93f73fc0f8c636fe60b81df975d5fc15b8c94f Jul 07 19:09:34.372589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.372506 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:34.473249 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.473216 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:34.647189 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.647073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:34.707850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.707820 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:34.821209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:34.821073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:35.016865 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:35.016836 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:35.112183 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:35.112138 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xwngj" event={"ID":"703f6de1-91b9-4fd0-8f74-c617e8968f09","Type":"ContainerStarted","Data":"9e735da4e271d08a9cee8af20f38e9a47ec44caa30a91c2b32e447e5ebf4fc03"} Jul 07 19:09:35.112340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:35.112214 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xwngj" event={"ID":"703f6de1-91b9-4fd0-8f74-c617e8968f09","Type":"ContainerStarted","Data":"d77750e81b670fc54b0ffeb7d909fdd336f64edd884b0cfba53f6e3a86132a05"} Jul 07 19:09:35.114571 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:35.114538 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" event={"ID":"69501978-25c4-4e80-bb45-009fdc2e3a84","Type":"ContainerStarted","Data":"d88c666b0f8da3bd3b5b72ce7bdb943690fbfdd91a8716609d755cb01de45d96"} Jul 07 19:09:35.115690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:35.115669 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" event={"ID":"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe","Type":"ContainerStarted","Data":"ea6a214db8273ed4a7180fb1da93f73fc0f8c636fe60b81df975d5fc15b8c94f"} Jul 07 19:09:35.134196 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:35.134153 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xwngj" podStartSLOduration=2.187597497 podStartE2EDuration="4.134138508s" totalImagesPullingTime="1.846947217s" totalInitContainerRuntime="99.593794ms" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:35.132371711 +0000 UTC m=+81.067997081" watchObservedRunningTime="2026-07-07 19:09:35.134138508 +0000 UTC m=+81.069763867" Jul 07 19:09:35.154695 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:35.154633 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mk2p6" podStartSLOduration=2.481837513 podStartE2EDuration="4.154617809s" totalImagesPullingTime="1.672780296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:35.152866053 +0000 UTC m=+81.088491414" watchObservedRunningTime="2026-07-07 19:09:35.154617809 +0000 UTC m=+81.090243171" Jul 07 19:09:36.070836 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.070806 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-698d7884fb-pvnc4"] Jul 07 19:09:36.074045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.074024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.077994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.077972 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 19:09:36.078110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.077997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 19:09:36.078110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.078015 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 19:09:36.078110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.078086 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 19:09:36.078110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.078100 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d69dopbhdrpe5\"" Jul 07 19:09:36.078340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.078327 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9w6g4\"" Jul 07 19:09:36.095928 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.095840 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-698d7884fb-pvnc4"] Jul 07 19:09:36.121044 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.121009 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerID="c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b" exitCode=0 Jul 07 19:09:36.121185 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.121102 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerDied","Data":"c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b"} Jul 07 19:09:36.163673 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.163635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-audit-log\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.164034 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.163974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-metrics-server-audit-profiles\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.164162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.164059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9jqj\" (UniqueName: \"kubernetes.io/projected/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-kube-api-access-n9jqj\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.164162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.164084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.164162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.164105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-client-ca-bundle\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.164341 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.164320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-secret-metrics-server-tls\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.164485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.164464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-secret-metrics-server-client-certs\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.264912 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.264852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9jqj\" (UniqueName: \"kubernetes.io/projected/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-kube-api-access-n9jqj\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.265114 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.264925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.265114 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.264946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-client-ca-bundle\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.265114 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.264989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-secret-metrics-server-tls\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.265430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.265110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-secret-metrics-server-client-certs\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.265430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.265150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-audit-log\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.265430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.265228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-metrics-server-audit-profiles\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.265793 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.265761 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.265906 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.265838 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-audit-log\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.266596 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.266572 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-metrics-server-audit-profiles\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.268186 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.268167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-secret-metrics-server-tls\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.268304 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.268258 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-secret-metrics-server-client-certs\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.268375 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.268354 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-client-ca-bundle\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.274418 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.274377 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9jqj\" (UniqueName: \"kubernetes.io/projected/f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df-kube-api-access-n9jqj\") pod \"metrics-server-698d7884fb-pvnc4\" (UID: \"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df\") " pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.384235 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.383912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:36.443186 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.443130 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:36.456829 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.456799 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8"] Jul 07 19:09:36.460229 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.460207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:36.463414 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.463237 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 19:09:36.463539 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.463516 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2shhc\"" Jul 07 19:09:36.472133 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.472104 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8"] Jul 07 19:09:36.535977 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.535924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:36.568117 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.568076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c37c8da-3a3a-4fa0-86b3-6c6d92807b40-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-jt8j8\" (UID: \"0c37c8da-3a3a-4fa0-86b3-6c6d92807b40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:36.568311 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.568159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6p9p\" (UniqueName: \"kubernetes.io/projected/0c37c8da-3a3a-4fa0-86b3-6c6d92807b40-kube-api-access-c6p9p\") pod \"monitoring-plugin-d5dcff7cf-jt8j8\" (UID: \"0c37c8da-3a3a-4fa0-86b3-6c6d92807b40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:36.669324 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.669302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c37c8da-3a3a-4fa0-86b3-6c6d92807b40-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-jt8j8\" (UID: \"0c37c8da-3a3a-4fa0-86b3-6c6d92807b40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:36.669441 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.669351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6p9p\" (UniqueName: \"kubernetes.io/projected/0c37c8da-3a3a-4fa0-86b3-6c6d92807b40-kube-api-access-c6p9p\") pod \"monitoring-plugin-d5dcff7cf-jt8j8\" (UID: \"0c37c8da-3a3a-4fa0-86b3-6c6d92807b40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:36.669502 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:36.669470 2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 07 19:09:36.669559 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:09:36.669549 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c37c8da-3a3a-4fa0-86b3-6c6d92807b40-monitoring-plugin-cert podName:0c37c8da-3a3a-4fa0-86b3-6c6d92807b40 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:37.169528038 +0000 UTC m=+83.105153400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0c37c8da-3a3a-4fa0-86b3-6c6d92807b40-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-jt8j8" (UID: "0c37c8da-3a3a-4fa0-86b3-6c6d92807b40") : secret "monitoring-plugin-cert" not found Jul 07 19:09:36.679831 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.679801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6p9p\" (UniqueName: \"kubernetes.io/projected/0c37c8da-3a3a-4fa0-86b3-6c6d92807b40-kube-api-access-c6p9p\") pod \"monitoring-plugin-d5dcff7cf-jt8j8\" (UID: \"0c37c8da-3a3a-4fa0-86b3-6c6d92807b40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:36.712154 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.712131 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-698d7884fb-pvnc4"] Jul 07 19:09:36.715191 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:36.715164 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a4c4c0_2376_4a6a_9eb5_dd01da1bf2df.slice/crio-63b86f434f74dfa087a5fe836d487ce1cc5f70402e6df51885c7ba7364b2ba62 WatchSource:0}: Error finding container 63b86f434f74dfa087a5fe836d487ce1cc5f70402e6df51885c7ba7364b2ba62: Status 404 returned error can't find the container with id 63b86f434f74dfa087a5fe836d487ce1cc5f70402e6df51885c7ba7364b2ba62 Jul 07 19:09:36.771328 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:36.771305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:37.127949 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.127913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" event={"ID":"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe","Type":"ContainerStarted","Data":"35e28223a6d117c8372b19e82e62dff97bf2a973453071bec499d11284a725fb"} Jul 07 19:09:37.128390 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.128004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" event={"ID":"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe","Type":"ContainerStarted","Data":"16a43f19b3e66b35f7b53f59409f9d5cfadb130f2bc68b40721c5769bb3b9219"} Jul 07 19:09:37.128390 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.128040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" event={"ID":"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe","Type":"ContainerStarted","Data":"e58b1b6df6c9adfa96d8863cabc1707c7ccda33f76ecc5f954b8609ecde42953"} Jul 07 19:09:37.129242 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.129212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" event={"ID":"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df","Type":"ContainerStarted","Data":"63b86f434f74dfa087a5fe836d487ce1cc5f70402e6df51885c7ba7364b2ba62"} Jul 07 19:09:37.174305 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.174271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c37c8da-3a3a-4fa0-86b3-6c6d92807b40-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-jt8j8\" (UID: \"0c37c8da-3a3a-4fa0-86b3-6c6d92807b40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:37.177149 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.177123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c37c8da-3a3a-4fa0-86b3-6c6d92807b40-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-jt8j8\" (UID: \"0c37c8da-3a3a-4fa0-86b3-6c6d92807b40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:37.345956 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.345922 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:37.372180 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.372157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:37.415852 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.415819 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:37.505837 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.505807 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8"] Jul 07 19:09:37.508577 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:37.508548 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c37c8da_3a3a_4fa0_86b3_6c6d92807b40.slice/crio-d63e4a458f47d843543ff0d761e6870745904422d01322c1c64e354355e68275 WatchSource:0}: Error finding container d63e4a458f47d843543ff0d761e6870745904422d01322c1c64e354355e68275: Status 404 returned error can't find the container with id d63e4a458f47d843543ff0d761e6870745904422d01322c1c64e354355e68275 Jul 07 19:09:37.511601 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.511581 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:37.609796 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:37.609767 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:38.098543 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.098519 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:09:38.104220 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.104195 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.111698 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.111677 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 19:09:38.112113 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.112096 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 19:09:38.112639 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.112603 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 19:09:38.113359 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.113338 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 19:09:38.113629 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.113612 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 19:09:38.115459 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.113998 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 19:09:38.115459 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.114454 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 19:09:38.115459 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.114810 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 19:09:38.115459 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.114824 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 19:09:38.115459 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.115062 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 19:09:38.120668 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.120605 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:38.124020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.124000 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 19:09:38.125613 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.125595 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 19:09:38.131738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.131721 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dlc2qgm2rf68q\"" Jul 07 19:09:38.132521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.131843 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lkqb8\"" Jul 07 19:09:38.132521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.132043 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 19:09:38.132666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.132598 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 19:09:38.136037 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.136009 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" event={"ID":"0c37c8da-3a3a-4fa0-86b3-6c6d92807b40","Type":"ContainerStarted","Data":"d63e4a458f47d843543ff0d761e6870745904422d01322c1c64e354355e68275"} Jul 07 19:09:38.156308 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.156284 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:09:38.184362 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.184472 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.184608 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.184676 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfm57\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-kube-api-access-vfm57\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.184736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.184736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.184838 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.184838 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.184838 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.184999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185305 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.185037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185305 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.185081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185305 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.185117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185305 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.185145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.185305 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.185169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286321 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286419 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286419 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286419 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfm57\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-kube-api-access-vfm57\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.286886 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.287550 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.286890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.289137 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.287772 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.289137 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.287818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.289137 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.288180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.289137 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.288555 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.289403 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.289333 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.289758 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.289732 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.294638 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.293761 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.294638 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.294595 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.296294 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.295715 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.296294 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.296210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.297803 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.296911 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.297803 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.296932 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.297803 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.296950 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.297803 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.297379 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.297803 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.297732 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.298692 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.298653 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.301430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.301355 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.302272 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.302229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.303134 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.303108 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.305184 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.305098 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfm57\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-kube-api-access-vfm57\") pod \"prometheus-k8s-0\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.372542 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.372509 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:38.378154 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.378094 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:09:38.428942 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.428912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:38.585386 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.585313 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:38.617493 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.617464 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:09:38.664564 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.664528 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:38.892310 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.892217 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:38.919013 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:09:38.918985 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45db9d6_9c7d_4c29_9502_f46b2bd6827c.slice/crio-5c5eb114ed5de04778460bcd0043c7b8a91d51af8a2e082f0ede9542e2fce23d WatchSource:0}: Error finding container 5c5eb114ed5de04778460bcd0043c7b8a91d51af8a2e082f0ede9542e2fce23d: Status 404 returned error can't find the container with id 5c5eb114ed5de04778460bcd0043c7b8a91d51af8a2e082f0ede9542e2fce23d Jul 07 19:09:38.958639 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:38.958605 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:39.148620 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:39.148539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerStarted","Data":"5c5eb114ed5de04778460bcd0043c7b8a91d51af8a2e082f0ede9542e2fce23d"} Jul 07 19:09:39.151911 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:39.151862 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" event={"ID":"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe","Type":"ContainerStarted","Data":"cacb90ed72e31db7448e4679b7fc9f2a04d74ffd33eebd832d10ebe345628e64"} Jul 07 19:09:39.152028 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:39.151942 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" event={"ID":"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe","Type":"ContainerStarted","Data":"1967e53b7fda3224a709fdb505a722a135aab95037656544714f9327f6673154"} Jul 07 19:09:39.152028 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:39.151972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" event={"ID":"ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe","Type":"ContainerStarted","Data":"ff7fea8ef90feba93655567ca062029738e501cec3042c38eacbe0a51fede84b"} Jul 07 19:09:39.180743 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:39.180693 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" podStartSLOduration=2.580258711 podStartE2EDuration="6.180672226s" totalImagesPullingTime="3.600413515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:39.179647073 +0000 UTC m=+85.115272435" watchObservedRunningTime="2026-07-07 19:09:39.180672226 +0000 UTC m=+85.116297588" Jul 07 19:09:39.812384 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:39.812360 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:39.912603 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:39.912553 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:40.162278 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.162193 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerStarted","Data":"172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed"} Jul 07 19:09:40.162661 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.162303 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerStarted","Data":"f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392"} Jul 07 19:09:40.162661 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.162350 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerStarted","Data":"ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf"} Jul 07 19:09:40.162661 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.162390 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerStarted","Data":"1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394"} Jul 07 19:09:40.162661 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.162428 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerStarted","Data":"2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299"} Jul 07 19:09:40.162661 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.162458 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerStarted","Data":"239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62"} Jul 07 19:09:40.163779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.163753 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" event={"ID":"0c37c8da-3a3a-4fa0-86b3-6c6d92807b40","Type":"ContainerStarted","Data":"b19ce5efccdc10ceebff36a73520726b1bc8a40103640be456f6deadf804fc02"} Jul 07 19:09:40.163930 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.163915 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:40.165430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.165402 2577 generic.go:356] "Generic (PLEG): container finished" podID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerID="b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81" exitCode=0 Jul 07 19:09:40.165536 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.165490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerDied","Data":"b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81"} Jul 07 19:09:40.167855 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.167253 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" event={"ID":"f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df","Type":"ContainerStarted","Data":"424c0c0861bb2e34e67d783172a676c20950e1384010facf1bb799f4f07a5156"} Jul 07 19:09:40.167855 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.167687 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:40.169524 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.169509 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" Jul 07 19:09:40.256244 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.256189 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.29969313 podStartE2EDuration="8.256169947s" totalImagesPullingTime="4.940114332s" totalInitContainerRuntime="16.362485ms" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:40.214762296 +0000 UTC m=+86.150387655" watchObservedRunningTime="2026-07-07 19:09:40.256169947 +0000 UTC m=+86.191795308" Jul 07 19:09:40.315220 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.315162 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-jt8j8" podStartSLOduration=2.170695216 podStartE2EDuration="4.315138027s" totalImagesPullingTime="2.144442811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:40.256800201 +0000 UTC m=+86.192425561" watchObservedRunningTime="2026-07-07 19:09:40.315138027 +0000 UTC m=+86.250763387" Jul 07 19:09:40.428121 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:40.428020 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" podStartSLOduration=1.494679944 podStartE2EDuration="4.428004616s" totalImagesPullingTime="2.933324672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:40.317392299 +0000 UTC m=+86.253017661" watchObservedRunningTime="2026-07-07 19:09:40.428004616 +0000 UTC m=+86.363629976" Jul 07 19:09:41.178372 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:41.178347 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bcbcfdd9b-n26x8" Jul 07 19:09:41.877889 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:41.877805 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:41.983439 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:41.983396 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:43.442588 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:43.442551 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:43.656808 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:43.656775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:44.277483 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:44.277454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:44.418996 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:44.418969 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:45.195275 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:45.195244 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerStarted","Data":"7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc"} Jul 07 19:09:45.195635 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:45.195353 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerStarted","Data":"6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d"} Jul 07 19:09:45.195635 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:45.195388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerStarted","Data":"d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875"} Jul 07 19:09:45.195635 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:45.195414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerStarted","Data":"9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598"} Jul 07 19:09:45.195635 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:45.195436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerStarted","Data":"45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96"} Jul 07 19:09:45.195635 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:45.195458 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerStarted","Data":"5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615"} Jul 07 19:09:45.248797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:45.248743 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.28506886 podStartE2EDuration="7.248726451s" totalImagesPullingTime="3.936434385s" totalInitContainerRuntime="27.223206ms" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:45.247930633 +0000 UTC m=+91.183555994" watchObservedRunningTime="2026-07-07 19:09:45.248726451 +0000 UTC m=+91.184351841" Jul 07 19:09:47.185420 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:47.185386 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:47.312651 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:47.312621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:47.459467 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:47.459398 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:47.935264 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:47.935238 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6dff4898-vncp9"] Jul 07 19:09:48.119799 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:48.119768 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:48.219789 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:48.219721 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:48.429098 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:48.429052 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:09:48.614823 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:48.614769 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:48.720391 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:48.720360 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:54.826300 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:54.826271 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:54.901415 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:54.901386 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:56.384015 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:56.383982 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:09:56.384384 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:09:56.384039 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:10:01.077122 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:01.077093 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wv6r9" Jul 07 19:10:09.828068 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:09.828038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:09.912289 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:09.912264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:12.956655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:12.956619 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7d6dff4898-vncp9" podUID="66d33a38-5a79-4b49-ba45-a06d28688968" containerName="console" containerID="cri-o://154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935" gracePeriod=15 Jul 07 19:10:13.194584 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.194563 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:10:13.286657 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.286633 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-trusted-ca-bundle\") pod \"66d33a38-5a79-4b49-ba45-a06d28688968\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " Jul 07 19:10:13.286782 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.286660 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-oauth-serving-cert\") pod \"66d33a38-5a79-4b49-ba45-a06d28688968\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " Jul 07 19:10:13.286782 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.286683 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-serving-cert\") pod \"66d33a38-5a79-4b49-ba45-a06d28688968\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " Jul 07 19:10:13.286782 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.286744 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-oauth-config\") pod \"66d33a38-5a79-4b49-ba45-a06d28688968\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " Jul 07 19:10:13.286955 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.286780 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-console-config\") pod \"66d33a38-5a79-4b49-ba45-a06d28688968\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " Jul 07 19:10:13.286955 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.286807 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-service-ca\") pod \"66d33a38-5a79-4b49-ba45-a06d28688968\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " Jul 07 19:10:13.286955 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.286838 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsk2j\" (UniqueName: \"kubernetes.io/projected/66d33a38-5a79-4b49-ba45-a06d28688968-kube-api-access-zsk2j\") pod \"66d33a38-5a79-4b49-ba45-a06d28688968\" (UID: \"66d33a38-5a79-4b49-ba45-a06d28688968\") " Jul 07 19:10:13.287140 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.287118 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66d33a38-5a79-4b49-ba45-a06d28688968" (UID: "66d33a38-5a79-4b49-ba45-a06d28688968"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:13.287202 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.287129 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "66d33a38-5a79-4b49-ba45-a06d28688968" (UID: "66d33a38-5a79-4b49-ba45-a06d28688968"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:13.287281 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.287260 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-service-ca" (OuterVolumeSpecName: "service-ca") pod "66d33a38-5a79-4b49-ba45-a06d28688968" (UID: "66d33a38-5a79-4b49-ba45-a06d28688968"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:13.287374 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.287345 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-console-config" (OuterVolumeSpecName: "console-config") pod "66d33a38-5a79-4b49-ba45-a06d28688968" (UID: "66d33a38-5a79-4b49-ba45-a06d28688968"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:13.289271 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.289251 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66d33a38-5a79-4b49-ba45-a06d28688968" (UID: "66d33a38-5a79-4b49-ba45-a06d28688968"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:13.289359 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.289281 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d33a38-5a79-4b49-ba45-a06d28688968-kube-api-access-zsk2j" (OuterVolumeSpecName: "kube-api-access-zsk2j") pod "66d33a38-5a79-4b49-ba45-a06d28688968" (UID: "66d33a38-5a79-4b49-ba45-a06d28688968"). InnerVolumeSpecName "kube-api-access-zsk2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:10:13.289359 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.289311 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66d33a38-5a79-4b49-ba45-a06d28688968" (UID: "66d33a38-5a79-4b49-ba45-a06d28688968"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:13.310570 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.310534 2577 generic.go:356] "Generic (PLEG): container finished" podID="66d33a38-5a79-4b49-ba45-a06d28688968" containerID="154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935" exitCode=0 Jul 07 19:10:13.310667 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.310582 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6dff4898-vncp9" Jul 07 19:10:13.310667 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.310617 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6dff4898-vncp9" event={"ID":"66d33a38-5a79-4b49-ba45-a06d28688968","Type":"ContainerDied","Data":"154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935"} Jul 07 19:10:13.310748 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.310684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6dff4898-vncp9" event={"ID":"66d33a38-5a79-4b49-ba45-a06d28688968","Type":"ContainerDied","Data":"576a278264a173584270dfc2175452ac863eeb3750f10c218468784ece168d99"} Jul 07 19:10:13.310748 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.310704 2577 scope.go:117] "RemoveContainer" containerID="154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935" Jul 07 19:10:13.319692 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.319676 2577 scope.go:117] "RemoveContainer" containerID="154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935" Jul 07 19:10:13.319956 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:13.319939 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935\": container with ID starting with 154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935 not found: ID does not exist" containerID="154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935" Jul 07 19:10:13.320022 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.319963 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935"} err="failed to get container status \"154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935\": rpc error: code = NotFound desc = could not find container \"154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935\": container with ID starting with 154b7fd7718648584f32b6548c5456aad1128e69a3cc3580e54dab074788c935 not found: ID does not exist" Jul 07 19:10:13.344919 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.344896 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6dff4898-vncp9"] Jul 07 19:10:13.346062 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.346046 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d6dff4898-vncp9"] Jul 07 19:10:13.387576 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.387558 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-oauth-config\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:13.387637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.387576 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-console-config\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:13.387637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.387586 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:13.387637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.387594 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsk2j\" (UniqueName: \"kubernetes.io/projected/66d33a38-5a79-4b49-ba45-a06d28688968-kube-api-access-zsk2j\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:13.387637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.387603 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-trusted-ca-bundle\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:13.387637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.387612 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d33a38-5a79-4b49-ba45-a06d28688968-oauth-serving-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:13.387637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:13.387620 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d33a38-5a79-4b49-ba45-a06d28688968-console-serving-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:14.602823 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:14.602786 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d33a38-5a79-4b49-ba45-a06d28688968" path="/var/lib/kubelet/pods/66d33a38-5a79-4b49-ba45-a06d28688968/volumes" Jul 07 19:10:16.390216 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:16.390187 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:10:16.393937 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:16.393913 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-698d7884fb-pvnc4" Jul 07 19:10:18.895542 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:18.895513 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:18.999142 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:18.999111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:24.900246 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:24.900213 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:24.974299 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:24.974265 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:29.330903 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:29.330856 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:38.430089 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:38.430053 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:38.454367 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:38.454342 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:39.429957 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:39.429929 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:50.973776 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:50.973741 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:51.173229 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:51.173136 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:52.073816 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.073784 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:10:52.074228 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.074060 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="alertmanager" containerID="cri-o://239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62" gracePeriod=120 Jul 07 19:10:52.074228 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.074133 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy-metric" containerID="cri-o://f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392" gracePeriod=120 Jul 07 19:10:52.074228 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.074163 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy-web" containerID="cri-o://1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394" gracePeriod=120 Jul 07 19:10:52.074228 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.074176 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="config-reloader" containerID="cri-o://2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299" gracePeriod=120 Jul 07 19:10:52.074465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.074204 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy" containerID="cri-o://ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf" gracePeriod=120 Jul 07 19:10:52.074465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.074235 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="prom-label-proxy" containerID="cri-o://172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed" gracePeriod=120 Jul 07 19:10:52.466791 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.466700 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerID="172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed" exitCode=0 Jul 07 19:10:52.466791 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.466726 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerID="ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf" exitCode=0 Jul 07 19:10:52.466791 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.466737 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerID="2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299" exitCode=0 Jul 07 19:10:52.466791 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.466745 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerID="239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62" exitCode=0 Jul 07 19:10:52.466791 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.466774 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerDied","Data":"172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed"} Jul 07 19:10:52.467076 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.466837 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerDied","Data":"ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf"} Jul 07 19:10:52.467076 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.466864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerDied","Data":"2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299"} Jul 07 19:10:52.467076 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:52.466913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerDied","Data":"239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62"} Jul 07 19:10:53.325755 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.325727 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.421797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421717 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-config-out\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.421797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421756 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.421797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421793 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-tls-assets\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421815 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zp5j\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-kube-api-access-2zp5j\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421832 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-web\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421848 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-config-volume\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421905 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-metrics-client-ca\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421932 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-main-db\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421961 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-main-tls\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.421997 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-web-config\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.422085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-cluster-tls-config\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422488 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.422114 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.422488 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.422140 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-trusted-ca-bundle\") pod \"0b4eef57-d363-4617-be33-6d0a5713bebb\" (UID: \"0b4eef57-d363-4617-be33-6d0a5713bebb\") " Jul 07 19:10:53.423127 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.423099 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:53.423390 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.423314 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:53.423496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.423331 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:10:53.424849 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.424787 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-kube-api-access-2zp5j" (OuterVolumeSpecName: "kube-api-access-2zp5j") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "kube-api-access-2zp5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:10:53.424849 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.424815 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-config-out" (OuterVolumeSpecName: "config-out") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:10:53.425296 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.425271 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:10:53.425572 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.425545 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:53.426038 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.426011 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:53.426135 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.426035 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:53.426277 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.426250 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-config-volume" (OuterVolumeSpecName: "config-volume") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:53.426545 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.426520 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:53.430492 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.430468 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:53.437082 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.437057 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-web-config" (OuterVolumeSpecName: "web-config") pod "0b4eef57-d363-4617-be33-6d0a5713bebb" (UID: "0b4eef57-d363-4617-be33-6d0a5713bebb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:53.473853 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.473827 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerID="f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392" exitCode=0 Jul 07 19:10:53.473853 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.473852 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerID="1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394" exitCode=0 Jul 07 19:10:53.474048 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.473913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerDied","Data":"f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392"} Jul 07 19:10:53.474048 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.473932 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.474048 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.474013 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerDied","Data":"1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394"} Jul 07 19:10:53.474048 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.474041 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b4eef57-d363-4617-be33-6d0a5713bebb","Type":"ContainerDied","Data":"6cab5f5a6a9dc4b0d9cc90ede75ba0d564df40edcb5632e9fd6c86f97b613c32"} Jul 07 19:10:53.474189 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.474072 2577 scope.go:117] "RemoveContainer" containerID="172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed" Jul 07 19:10:53.482592 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.482577 2577 scope.go:117] "RemoveContainer" containerID="f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392" Jul 07 19:10:53.489556 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.489542 2577 scope.go:117] "RemoveContainer" containerID="ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf" Jul 07 19:10:53.496056 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.496038 2577 scope.go:117] "RemoveContainer" containerID="1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394" Jul 07 19:10:53.502024 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.502004 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:10:53.503133 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.503119 2577 scope.go:117] "RemoveContainer" containerID="2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299" Jul 07 19:10:53.508964 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.508944 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:10:53.510671 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.510655 2577 scope.go:117] "RemoveContainer" containerID="239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62" Jul 07 19:10:53.517222 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.517208 2577 scope.go:117] "RemoveContainer" containerID="c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b" Jul 07 19:10:53.523448 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523430 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-tls-assets\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523450 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zp5j\" (UniqueName: \"kubernetes.io/projected/0b4eef57-d363-4617-be33-6d0a5713bebb-kube-api-access-2zp5j\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523462 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523473 2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-config-volume\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523481 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-metrics-client-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523489 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-main-db\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523497 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-main-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523505 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-web-config\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523516 2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-cluster-tls-config\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523529 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523541 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4eef57-d363-4617-be33-6d0a5713bebb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523550 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b4eef57-d363-4617-be33-6d0a5713bebb-config-out\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523558 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b4eef57-d363-4617-be33-6d0a5713bebb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:53.523989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523658 2577 scope.go:117] "RemoveContainer" containerID="172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed" Jul 07 19:10:53.523989 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:53.523911 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed\": container with ID starting with 172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed not found: ID does not exist" containerID="172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed" Jul 07 19:10:53.523989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523935 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed"} err="failed to get container status \"172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed\": rpc error: code = NotFound desc = could not find container \"172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed\": container with ID starting with 172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed not found: ID does not exist" Jul 07 19:10:53.523989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.523956 2577 scope.go:117] "RemoveContainer" containerID="f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392" Jul 07 19:10:53.524253 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:53.524198 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392\": container with ID starting with f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392 not found: ID does not exist" containerID="f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392" Jul 07 19:10:53.524253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.524215 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392"} err="failed to get container status \"f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392\": rpc error: code = NotFound desc = could not find container \"f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392\": container with ID starting with f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392 not found: ID does not exist" Jul 07 19:10:53.524253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.524228 2577 scope.go:117] "RemoveContainer" containerID="ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf" Jul 07 19:10:53.524475 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:53.524458 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf\": container with ID starting with ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf not found: ID does not exist" containerID="ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf" Jul 07 19:10:53.524560 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.524484 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf"} err="failed to get container status \"ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf\": rpc error: code = NotFound desc = could not find container \"ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf\": container with ID starting with ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf not found: ID does not exist" Jul 07 19:10:53.524560 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.524506 2577 scope.go:117] "RemoveContainer" containerID="1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394" Jul 07 19:10:53.524775 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:53.524759 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394\": container with ID starting with 1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394 not found: ID does not exist" containerID="1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394" Jul 07 19:10:53.524824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.524779 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394"} err="failed to get container status \"1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394\": rpc error: code = NotFound desc = could not find container \"1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394\": container with ID starting with 1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394 not found: ID does not exist" Jul 07 19:10:53.524824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.524792 2577 scope.go:117] "RemoveContainer" containerID="2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299" Jul 07 19:10:53.525034 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:53.525018 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299\": container with ID starting with 2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299 not found: ID does not exist" containerID="2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299" Jul 07 19:10:53.525081 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525039 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299"} err="failed to get container status \"2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299\": rpc error: code = NotFound desc = could not find container \"2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299\": container with ID starting with 2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299 not found: ID does not exist" Jul 07 19:10:53.525081 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525052 2577 scope.go:117] "RemoveContainer" containerID="239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62" Jul 07 19:10:53.525263 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:53.525249 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62\": container with ID starting with 239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62 not found: ID does not exist" containerID="239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62" Jul 07 19:10:53.525311 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525265 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62"} err="failed to get container status \"239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62\": rpc error: code = NotFound desc = could not find container \"239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62\": container with ID starting with 239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62 not found: ID does not exist" Jul 07 19:10:53.525311 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525276 2577 scope.go:117] "RemoveContainer" containerID="c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b" Jul 07 19:10:53.525493 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:53.525478 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b\": container with ID starting with c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b not found: ID does not exist" containerID="c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b" Jul 07 19:10:53.525533 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525495 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b"} err="failed to get container status \"c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b\": rpc error: code = NotFound desc = could not find container \"c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b\": container with ID starting with c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b not found: ID does not exist" Jul 07 19:10:53.525533 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525506 2577 scope.go:117] "RemoveContainer" containerID="172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed" Jul 07 19:10:53.525705 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525688 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed"} err="failed to get container status \"172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed\": rpc error: code = NotFound desc = could not find container \"172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed\": container with ID starting with 172513c536e49494acb31f5f71144d9a17e98664994455282b05c674d00aabed not found: ID does not exist" Jul 07 19:10:53.525743 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525707 2577 scope.go:117] "RemoveContainer" containerID="f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392" Jul 07 19:10:53.525932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525912 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392"} err="failed to get container status \"f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392\": rpc error: code = NotFound desc = could not find container \"f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392\": container with ID starting with f9c1de436fd45433b607a8b8f601c8dcff4f2c597f73d6439d72cee1f471e392 not found: ID does not exist" Jul 07 19:10:53.525976 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.525933 2577 scope.go:117] "RemoveContainer" containerID="ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf" Jul 07 19:10:53.526147 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.526132 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf"} err="failed to get container status \"ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf\": rpc error: code = NotFound desc = could not find container \"ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf\": container with ID starting with ff0f665d380b621fcea89e623acfe557192eb8fb937905ea716f713fc61c80bf not found: ID does not exist" Jul 07 19:10:53.526190 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.526148 2577 scope.go:117] "RemoveContainer" containerID="1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394" Jul 07 19:10:53.526330 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.526314 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394"} err="failed to get container status \"1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394\": rpc error: code = NotFound desc = could not find container \"1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394\": container with ID starting with 1f470f02dd5c1797783bd6482811c7cdd29081786a935ecab28966c44035b394 not found: ID does not exist" Jul 07 19:10:53.526365 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.526330 2577 scope.go:117] "RemoveContainer" containerID="2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299" Jul 07 19:10:53.526502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.526486 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299"} err="failed to get container status \"2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299\": rpc error: code = NotFound desc = could not find container \"2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299\": container with ID starting with 2d7e3778ff0a2bbe3b86a029fdff6ed58abb9b8782c3352a30a550e5fa05a299 not found: ID does not exist" Jul 07 19:10:53.526542 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.526503 2577 scope.go:117] "RemoveContainer" containerID="239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62" Jul 07 19:10:53.526679 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.526664 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62"} err="failed to get container status \"239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62\": rpc error: code = NotFound desc = could not find container \"239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62\": container with ID starting with 239886b7dc6f51f048d34e60e7cc73b3de900e73f5904a779a504a638a022a62 not found: ID does not exist" Jul 07 19:10:53.526721 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.526679 2577 scope.go:117] "RemoveContainer" containerID="c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b" Jul 07 19:10:53.526900 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.526864 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b"} err="failed to get container status \"c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b\": rpc error: code = NotFound desc = could not find container \"c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b\": container with ID starting with c3d0014185215c1def5bd94763a00c30ada353472143d28f099f93119c5ee48b not found: ID does not exist" Jul 07 19:10:53.542629 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.542609 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:10:53.542971 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.542957 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="config-reloader" Jul 07 19:10:53.543017 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.542974 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="config-reloader" Jul 07 19:10:53.543017 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.542985 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="alertmanager" Jul 07 19:10:53.543017 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.542994 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="alertmanager" Jul 07 19:10:53.543017 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543009 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy-web" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543018 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy-web" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543029 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d33a38-5a79-4b49-ba45-a06d28688968" containerName="console" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543035 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d33a38-5a79-4b49-ba45-a06d28688968" containerName="console" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543041 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543046 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543055 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="init-config-reloader" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543060 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="init-config-reloader" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543066 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy-metric" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543071 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy-metric" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543089 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="prom-label-proxy" Jul 07 19:10:53.543132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543099 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="prom-label-proxy" Jul 07 19:10:53.543443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543159 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="config-reloader" Jul 07 19:10:53.543443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543172 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy-web" Jul 07 19:10:53.543443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543182 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="alertmanager" Jul 07 19:10:53.543443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543187 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="prom-label-proxy" Jul 07 19:10:53.543443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543193 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy" Jul 07 19:10:53.543443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543198 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d33a38-5a79-4b49-ba45-a06d28688968" containerName="console" Jul 07 19:10:53.543443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.543204 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" containerName="kube-rbac-proxy-metric" Jul 07 19:10:53.546691 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.546675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.550354 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.549764 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 19:10:53.550354 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.549817 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mqthr\"" Jul 07 19:10:53.550354 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.549768 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 19:10:53.550354 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.550086 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 19:10:53.550354 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.550113 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 19:10:53.550354 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.550145 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 19:10:53.550354 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.550146 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 19:10:53.550723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.550406 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 19:10:53.550781 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.550744 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 19:10:53.555351 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.555321 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 19:10:53.566653 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.566629 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:10:53.624766 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.624727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3d4b3b5-a308-4726-a62c-e5c2f891e671-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.624977 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.624823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.624977 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.624897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3d4b3b5-a308-4726-a62c-e5c2f891e671-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.624977 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.624945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.625110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.624982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.625110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.625003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d4b3b5-a308-4726-a62c-e5c2f891e671-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.625110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.625040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d4b3b5-a308-4726-a62c-e5c2f891e671-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.625110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.625062 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.625110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.625076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.625110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.625095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3d4b3b5-a308-4726-a62c-e5c2f891e671-config-out\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.625312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.625120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-web-config\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.625312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.625167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrnds\" (UniqueName: \"kubernetes.io/projected/f3d4b3b5-a308-4726-a62c-e5c2f891e671-kube-api-access-nrnds\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.625312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.625191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.725774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.725695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrnds\" (UniqueName: \"kubernetes.io/projected/f3d4b3b5-a308-4726-a62c-e5c2f891e671-kube-api-access-nrnds\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.725774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.725727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.725779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3d4b3b5-a308-4726-a62c-e5c2f891e671-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.725806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3d4b3b5-a308-4726-a62c-e5c2f891e671-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d4b3b5-a308-4726-a62c-e5c2f891e671-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d4b3b5-a308-4726-a62c-e5c2f891e671-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726985 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726985 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3d4b3b5-a308-4726-a62c-e5c2f891e671-config-out\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726985 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-web-config\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.726985 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.726493 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3d4b3b5-a308-4726-a62c-e5c2f891e671-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.727793 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.727706 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d4b3b5-a308-4726-a62c-e5c2f891e671-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.727793 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.727744 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d4b3b5-a308-4726-a62c-e5c2f891e671-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.729145 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.729121 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.729145 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.729132 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3d4b3b5-a308-4726-a62c-e5c2f891e671-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.729306 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.729288 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-web-config\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.729481 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.729454 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.729911 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.729863 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.730008 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.729929 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.730201 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.730182 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.730984 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.730961 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3d4b3b5-a308-4726-a62c-e5c2f891e671-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.731167 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.731153 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3d4b3b5-a308-4726-a62c-e5c2f891e671-config-out\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.735639 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.735624 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrnds\" (UniqueName: \"kubernetes.io/projected/f3d4b3b5-a308-4726-a62c-e5c2f891e671-kube-api-access-nrnds\") pod \"alertmanager-main-0\" (UID: \"f3d4b3b5-a308-4726-a62c-e5c2f891e671\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:53.857223 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:53.857175 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:54.012342 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:54.012320 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:10:54.014348 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:10:54.014320 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d4b3b5_a308_4726_a62c_e5c2f891e671.slice/crio-613545245a98a794f6e5e781bad6ca049e27c3add6fd40682d9bd5a18ce40925 WatchSource:0}: Error finding container 613545245a98a794f6e5e781bad6ca049e27c3add6fd40682d9bd5a18ce40925: Status 404 returned error can't find the container with id 613545245a98a794f6e5e781bad6ca049e27c3add6fd40682d9bd5a18ce40925 Jul 07 19:10:54.478817 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:54.478734 2577 generic.go:356] "Generic (PLEG): container finished" podID="f3d4b3b5-a308-4726-a62c-e5c2f891e671" containerID="05a780718eb970baf0def47b3790087eea92866c7c41991c8ba5cb478f81d167" exitCode=0 Jul 07 19:10:54.478817 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:54.478776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d4b3b5-a308-4726-a62c-e5c2f891e671","Type":"ContainerDied","Data":"05a780718eb970baf0def47b3790087eea92866c7c41991c8ba5cb478f81d167"} Jul 07 19:10:54.479273 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:54.478848 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d4b3b5-a308-4726-a62c-e5c2f891e671","Type":"ContainerStarted","Data":"613545245a98a794f6e5e781bad6ca049e27c3add6fd40682d9bd5a18ce40925"} Jul 07 19:10:54.607919 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:54.604970 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b4eef57-d363-4617-be33-6d0a5713bebb" path="/var/lib/kubelet/pods/0b4eef57-d363-4617-be33-6d0a5713bebb/volumes" Jul 07 19:10:55.487513 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:55.487479 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d4b3b5-a308-4726-a62c-e5c2f891e671","Type":"ContainerStarted","Data":"02f0131b51f2a777cb4c357b3d8b41bdd272139b07a7a3b965cc115680af9d17"} Jul 07 19:10:55.487913 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:55.487540 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d4b3b5-a308-4726-a62c-e5c2f891e671","Type":"ContainerStarted","Data":"c08f76a8323b1f549225965b793793d8f7259fb9c1a1f60deef48a319d7a3b08"} Jul 07 19:10:55.487913 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:55.487564 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d4b3b5-a308-4726-a62c-e5c2f891e671","Type":"ContainerStarted","Data":"2db9ccce865b27e5061a99f6ee66a4db3d031da2552a97189cf31ebc987c2082"} Jul 07 19:10:55.487913 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:55.487589 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d4b3b5-a308-4726-a62c-e5c2f891e671","Type":"ContainerStarted","Data":"b2dfd18d4c12fd7dde6d10013d9a0102614ec4e2310fe89af2bba4deac487645"} Jul 07 19:10:55.487913 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:55.487613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d4b3b5-a308-4726-a62c-e5c2f891e671","Type":"ContainerStarted","Data":"5f3ad9995a8dbdafcd29009190f268f0a31a477761162d44fe8195ccd9e5b50b"} Jul 07 19:10:55.487913 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:55.487637 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d4b3b5-a308-4726-a62c-e5c2f891e671","Type":"ContainerStarted","Data":"7c25ed0862495324d3e79bdc3c6578f25054b7e8eea2e1127957cb512f17d883"} Jul 07 19:10:55.525333 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:55.525266 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.510525763 podStartE2EDuration="2.525245282s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.719519ms" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:55.522960822 +0000 UTC m=+161.458586181" watchObservedRunningTime="2026-07-07 19:10:55.525245282 +0000 UTC m=+161.460870643" Jul 07 19:10:56.100255 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.100219 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-bbdd4f87-jfknr"] Jul 07 19:10:56.103173 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.103148 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.109699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.109674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 19:10:56.109822 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.109704 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 19:10:56.109822 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.109674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 19:10:56.110999 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.110800 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 19:10:56.110999 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.110923 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 19:10:56.110999 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.110959 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7nn9j\"" Jul 07 19:10:56.118672 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.118645 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 19:10:56.125569 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.125543 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bbdd4f87-jfknr"] Jul 07 19:10:56.247786 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.247747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-telemeter-client-tls\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.247961 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.247791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.247961 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.247819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9x7g\" (UniqueName: \"kubernetes.io/projected/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-kube-api-access-k9x7g\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.247961 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.247914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-serving-certs-ca-bundle\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.248099 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.247982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-secret-telemeter-client\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.248099 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.248052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-metrics-client-ca\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.248176 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.248098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-federate-client-tls\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.248176 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.248147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.349030 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.348997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-metrics-client-ca\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.349283 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.349041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-federate-client-tls\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.349283 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.349065 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.349283 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.349110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-telemeter-client-tls\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.349283 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.349135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.349283 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.349161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9x7g\" (UniqueName: \"kubernetes.io/projected/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-kube-api-access-k9x7g\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.349283 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.349186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-serving-certs-ca-bundle\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.349283 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.349224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-secret-telemeter-client\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.350132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.350107 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-metrics-client-ca\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.350414 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.350364 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-serving-certs-ca-bundle\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.350531 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.350512 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.351947 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.351923 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-telemeter-client-tls\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.352051 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.351961 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-federate-client-tls\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.352110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.352068 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.352207 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.352191 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-secret-telemeter-client\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.361051 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.361032 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9x7g\" (UniqueName: \"kubernetes.io/projected/2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b-kube-api-access-k9x7g\") pod \"telemeter-client-bbdd4f87-jfknr\" (UID: \"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b\") " pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.413741 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.413467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" Jul 07 19:10:56.416726 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.416578 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:10:56.416943 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.416911 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="prometheus" containerID="cri-o://7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc" gracePeriod=600 Jul 07 19:10:56.417093 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.416935 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy-web" containerID="cri-o://9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598" gracePeriod=600 Jul 07 19:10:56.417093 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.416947 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d" gracePeriod=600 Jul 07 19:10:56.417093 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.417063 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy" containerID="cri-o://d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875" gracePeriod=600 Jul 07 19:10:56.417093 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.417070 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="thanos-sidecar" containerID="cri-o://45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96" gracePeriod=600 Jul 07 19:10:56.417293 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.417091 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="config-reloader" containerID="cri-o://5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615" gracePeriod=600 Jul 07 19:10:56.590091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:56.590050 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bbdd4f87-jfknr"] Jul 07 19:10:56.592174 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:10:56.592146 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0a1d1e_f902_47bf_8b9a_ff2ccf1ce88b.slice/crio-63635f89e798c2d549d507a17e0df9e84649288340028e928a3f8fbde33f7c65 WatchSource:0}: Error finding container 63635f89e798c2d549d507a17e0df9e84649288340028e928a3f8fbde33f7c65: Status 404 returned error can't find the container with id 63635f89e798c2d549d507a17e0df9e84649288340028e928a3f8fbde33f7c65 Jul 07 19:10:57.505318 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505288 2577 generic.go:356] "Generic (PLEG): container finished" podID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerID="6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d" exitCode=0 Jul 07 19:10:57.505318 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505318 2577 generic.go:356] "Generic (PLEG): container finished" podID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerID="d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875" exitCode=0 Jul 07 19:10:57.505507 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505327 2577 generic.go:356] "Generic (PLEG): container finished" podID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerID="45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96" exitCode=0 Jul 07 19:10:57.505507 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505335 2577 generic.go:356] "Generic (PLEG): container finished" podID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerID="5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615" exitCode=0 Jul 07 19:10:57.505507 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505343 2577 generic.go:356] "Generic (PLEG): container finished" podID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerID="7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc" exitCode=0 Jul 07 19:10:57.505507 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerDied","Data":"6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d"} Jul 07 19:10:57.505507 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505459 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerDied","Data":"d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875"} Jul 07 19:10:57.505507 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerDied","Data":"45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96"} Jul 07 19:10:57.505759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505544 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerDied","Data":"5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615"} Jul 07 19:10:57.505759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.505586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerDied","Data":"7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc"} Jul 07 19:10:57.506789 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.506767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" event={"ID":"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b","Type":"ContainerStarted","Data":"63635f89e798c2d549d507a17e0df9e84649288340028e928a3f8fbde33f7c65"} Jul 07 19:10:57.673100 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.673075 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:57.764387 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764348 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-tls-assets\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.764571 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764400 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.764571 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764436 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-0\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.764686 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764605 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-tls\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.764686 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764652 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-thanos-prometheus-http-client-file\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.764686 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764681 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-kube-rbac-proxy\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.764828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764716 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-2\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.764828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764740 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-1\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.764828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764765 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-serving-certs-ca-bundle\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.764828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764798 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-web-config\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764833 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764865 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764911 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfm57\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-kube-api-access-vfm57\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764955 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-metrics-client-certs\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.764979 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-grpc-tls\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.765010 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-metrics-client-ca\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.765059 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-db\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.765090 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-kubelet-serving-ca-bundle\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.765123 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config-out\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.765326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.765152 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-trusted-ca-bundle\") pod \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\" (UID: \"b45db9d6-9c7d-4c29-9502-f46b2bd6827c\") " Jul 07 19:10:57.768474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.765905 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:57.768474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.766150 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:57.768474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.766791 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:57.768474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.766833 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:57.768474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.767203 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:57.768474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.767383 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:57.768474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.767479 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:57.768474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.767955 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:10:57.768474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.768432 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:10:57.769196 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.769168 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:10:57.769429 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.769385 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:57.770395 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.770365 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config" (OuterVolumeSpecName: "config") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:57.770516 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.770488 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config-out" (OuterVolumeSpecName: "config-out") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:10:57.770700 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.770681 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:57.770803 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.770766 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:57.770862 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.770814 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:57.771321 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.771297 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-kube-api-access-vfm57" (OuterVolumeSpecName: "kube-api-access-vfm57") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "kube-api-access-vfm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:10:57.771689 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.771664 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:57.772768 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.772724 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:57.784186 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.784160 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-web-config" (OuterVolumeSpecName: "web-config") pod "b45db9d6-9c7d-4c29-9502-f46b2bd6827c" (UID: "b45db9d6-9c7d-4c29-9502-f46b2bd6827c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:10:57.866947 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.866912 2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-metrics-client-certs\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.866947 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.866943 2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-grpc-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.866947 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.866956 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-metrics-client-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.866968 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-db\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.866982 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.866994 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config-out\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867004 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867015 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-tls-assets\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867027 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867039 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867050 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867062 2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867074 2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-kube-rbac-proxy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867085 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867098 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867110 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867126 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-web-config\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867138 2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-config\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867153 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:57.867212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:57.867166 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfm57\" (UniqueName: \"kubernetes.io/projected/b45db9d6-9c7d-4c29-9502-f46b2bd6827c-kube-api-access-vfm57\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:10:58.516141 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.516103 2577 generic.go:356] "Generic (PLEG): container finished" podID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerID="9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598" exitCode=0 Jul 07 19:10:58.516141 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.516147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerDied","Data":"9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598"} Jul 07 19:10:58.516370 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.516201 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.516370 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.516217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b45db9d6-9c7d-4c29-9502-f46b2bd6827c","Type":"ContainerDied","Data":"5c5eb114ed5de04778460bcd0043c7b8a91d51af8a2e082f0ede9542e2fce23d"} Jul 07 19:10:58.516370 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.516268 2577 scope.go:117] "RemoveContainer" containerID="6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d" Jul 07 19:10:58.524563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.524537 2577 scope.go:117] "RemoveContainer" containerID="d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875" Jul 07 19:10:58.542813 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.542784 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:10:58.548109 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.548084 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:10:58.587256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587223 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:10:58.587574 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587550 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="config-reloader" Jul 07 19:10:58.587574 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587569 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="config-reloader" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587580 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587588 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587606 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="prometheus" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587613 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="prometheus" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587624 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="thanos-sidecar" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587631 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="thanos-sidecar" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587644 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy-web" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587651 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy-web" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587663 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy-thanos" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587671 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy-thanos" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587688 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="init-config-reloader" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587696 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="init-config-reloader" Jul 07 19:10:58.587774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587771 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy-thanos" Jul 07 19:10:58.588395 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587785 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="prometheus" Jul 07 19:10:58.588395 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587794 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="config-reloader" Jul 07 19:10:58.588395 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587804 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy" Jul 07 19:10:58.588395 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587813 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="thanos-sidecar" Jul 07 19:10:58.588395 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.587824 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" containerName="kube-rbac-proxy-web" Jul 07 19:10:58.591334 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.591310 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.594454 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.594402 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 19:10:58.594454 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.594422 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 19:10:58.594667 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.594578 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 19:10:58.594799 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.594687 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lkqb8\"" Jul 07 19:10:58.594943 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.594922 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 19:10:58.595066 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.595046 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 19:10:58.595115 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.595065 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 19:10:58.595115 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.595080 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 19:10:58.595115 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.595087 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 19:10:58.595425 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.595369 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 19:10:58.595948 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.595823 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dlc2qgm2rf68q\"" Jul 07 19:10:58.595948 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.595849 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 19:10:58.596365 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.596344 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 19:10:58.596499 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.596479 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 19:10:58.597397 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.597371 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 19:10:58.604569 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.604541 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b45db9d6-9c7d-4c29-9502-f46b2bd6827c" path="/var/lib/kubelet/pods/b45db9d6-9c7d-4c29-9502-f46b2bd6827c/volumes" Jul 07 19:10:58.604932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.604912 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 19:10:58.613980 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.613928 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:10:58.673365 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e88c4bd6-8796-4ba1-8527-ad3677782c0f-config-out\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-web-config\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673616 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lpdd\" (UniqueName: \"kubernetes.io/projected/e88c4bd6-8796-4ba1-8527-ad3677782c0f-kube-api-access-5lpdd\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.673850 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-config\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.674475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.674475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.674475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.673968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.674475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.674020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.674475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.674046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.674475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.674077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e88c4bd6-8796-4ba1-8527-ad3677782c0f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.674475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.674093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.774816 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.774786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lpdd\" (UniqueName: \"kubernetes.io/projected/e88c4bd6-8796-4ba1-8527-ad3677782c0f-kube-api-access-5lpdd\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.774816 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.774822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-config\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775036 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.774848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775036 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.774898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775036 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.774926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775330 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775425 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775425 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e88c4bd6-8796-4ba1-8527-ad3677782c0f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775425 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775958 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e88c4bd6-8796-4ba1-8527-ad3677782c0f-config-out\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775958 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-web-config\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775958 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775958 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775958 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775958 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775958 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.775958 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.775764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.777054 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.776752 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.778239 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.778213 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-config\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.778539 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.778511 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.778622 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.778548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.778676 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.778612 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.779145 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.779124 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.779301 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.779274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.779301 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.779296 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.779838 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.779815 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.780035 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.780005 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.780184 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.780156 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.780568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.780541 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e88c4bd6-8796-4ba1-8527-ad3677782c0f-config-out\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.781096 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.781072 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.781303 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.781278 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88c4bd6-8796-4ba1-8527-ad3677782c0f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.781502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.781480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.782040 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.781999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-web-config\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.782470 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.782448 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e88c4bd6-8796-4ba1-8527-ad3677782c0f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.783350 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.783328 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e88c4bd6-8796-4ba1-8527-ad3677782c0f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.784759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.784739 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lpdd\" (UniqueName: \"kubernetes.io/projected/e88c4bd6-8796-4ba1-8527-ad3677782c0f-kube-api-access-5lpdd\") pod \"prometheus-k8s-0\" (UID: \"e88c4bd6-8796-4ba1-8527-ad3677782c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.904194 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.904113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:10:58.974403 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:58.974377 2577 scope.go:117] "RemoveContainer" containerID="9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598" Jul 07 19:10:59.002776 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.002747 2577 scope.go:117] "RemoveContainer" containerID="45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96" Jul 07 19:10:59.010313 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.010295 2577 scope.go:117] "RemoveContainer" containerID="5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615" Jul 07 19:10:59.018897 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.018857 2577 scope.go:117] "RemoveContainer" containerID="7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc" Jul 07 19:10:59.028573 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.028551 2577 scope.go:117] "RemoveContainer" containerID="b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81" Jul 07 19:10:59.038638 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.038618 2577 scope.go:117] "RemoveContainer" containerID="6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d" Jul 07 19:10:59.038988 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:59.038954 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d\": container with ID starting with 6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d not found: ID does not exist" containerID="6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d" Jul 07 19:10:59.039083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.038997 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d"} err="failed to get container status \"6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d\": rpc error: code = NotFound desc = could not find container \"6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d\": container with ID starting with 6cec1114fc67e0edb436f083923658f07037f1de6933f9131f8de708cddd857d not found: ID does not exist" Jul 07 19:10:59.039083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.039017 2577 scope.go:117] "RemoveContainer" containerID="d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875" Jul 07 19:10:59.039310 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:59.039289 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875\": container with ID starting with d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875 not found: ID does not exist" containerID="d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875" Jul 07 19:10:59.039422 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.039317 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875"} err="failed to get container status \"d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875\": rpc error: code = NotFound desc = could not find container \"d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875\": container with ID starting with d8b722f36c217ddbe66edb653362f0b6fb70fcb7afa22e948d8eecdd913e9875 not found: ID does not exist" Jul 07 19:10:59.039422 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.039337 2577 scope.go:117] "RemoveContainer" containerID="9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598" Jul 07 19:10:59.039618 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:59.039600 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598\": container with ID starting with 9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598 not found: ID does not exist" containerID="9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598" Jul 07 19:10:59.039680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.039627 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598"} err="failed to get container status \"9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598\": rpc error: code = NotFound desc = could not find container \"9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598\": container with ID starting with 9b4a3481b7651c13bc09b837a3d657c577b2b2ed8cd93ca595c0e5d9fa3b4598 not found: ID does not exist" Jul 07 19:10:59.039680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.039648 2577 scope.go:117] "RemoveContainer" containerID="45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96" Jul 07 19:10:59.039929 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:59.039909 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96\": container with ID starting with 45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96 not found: ID does not exist" containerID="45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96" Jul 07 19:10:59.040010 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.039933 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96"} err="failed to get container status \"45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96\": rpc error: code = NotFound desc = could not find container \"45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96\": container with ID starting with 45aa023fe95913552756d1de4f92071839abb8a2faa107ebfe452fde25a88c96 not found: ID does not exist" Jul 07 19:10:59.040010 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.039947 2577 scope.go:117] "RemoveContainer" containerID="5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615" Jul 07 19:10:59.040216 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:59.040194 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615\": container with ID starting with 5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615 not found: ID does not exist" containerID="5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615" Jul 07 19:10:59.040293 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.040220 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615"} err="failed to get container status \"5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615\": rpc error: code = NotFound desc = could not find container \"5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615\": container with ID starting with 5003fe6afa87164ccd479a2d74dbb51ebc80959a99d10dbeeb85a9a25d7b8615 not found: ID does not exist" Jul 07 19:10:59.040293 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.040235 2577 scope.go:117] "RemoveContainer" containerID="7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc" Jul 07 19:10:59.040451 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:59.040436 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc\": container with ID starting with 7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc not found: ID does not exist" containerID="7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc" Jul 07 19:10:59.040498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.040455 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc"} err="failed to get container status \"7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc\": rpc error: code = NotFound desc = could not find container \"7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc\": container with ID starting with 7236caf30024a5f2f2b41a00d39f3efa7aabe4fb8e493a07905e0a6a283034cc not found: ID does not exist" Jul 07 19:10:59.040498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.040467 2577 scope.go:117] "RemoveContainer" containerID="b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81" Jul 07 19:10:59.040683 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:10:59.040667 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81\": container with ID starting with b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81 not found: ID does not exist" containerID="b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81" Jul 07 19:10:59.040720 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.040686 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81"} err="failed to get container status \"b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81\": rpc error: code = NotFound desc = could not find container \"b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81\": container with ID starting with b51cf00ddcec0e9286d5662dd357195fb560db11172efb67b8e3177f3efbcd81 not found: ID does not exist" Jul 07 19:10:59.121360 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.121338 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:10:59.122841 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:10:59.122815 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88c4bd6_8796_4ba1_8527_ad3677782c0f.slice/crio-ea93b99f582d01728ceced7b3286d26401314a051e9efb3a2c0e691bc0e778f9 WatchSource:0}: Error finding container ea93b99f582d01728ceced7b3286d26401314a051e9efb3a2c0e691bc0e778f9: Status 404 returned error can't find the container with id ea93b99f582d01728ceced7b3286d26401314a051e9efb3a2c0e691bc0e778f9 Jul 07 19:10:59.521584 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.521546 2577 generic.go:356] "Generic (PLEG): container finished" podID="e88c4bd6-8796-4ba1-8527-ad3677782c0f" containerID="73b5dc7230f389199338f1c5cfef48e132ac09c7e99b5ec37cf37cb54f1b026c" exitCode=0 Jul 07 19:10:59.521741 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.521640 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e88c4bd6-8796-4ba1-8527-ad3677782c0f","Type":"ContainerDied","Data":"73b5dc7230f389199338f1c5cfef48e132ac09c7e99b5ec37cf37cb54f1b026c"} Jul 07 19:10:59.521741 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.521730 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e88c4bd6-8796-4ba1-8527-ad3677782c0f","Type":"ContainerStarted","Data":"ea93b99f582d01728ceced7b3286d26401314a051e9efb3a2c0e691bc0e778f9"} Jul 07 19:10:59.524781 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.524761 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" event={"ID":"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b","Type":"ContainerStarted","Data":"0a73b3ee146e7456b5624ba1c95c9be35a703d4e69803686405016b7039755ba"} Jul 07 19:10:59.524901 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.524803 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" event={"ID":"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b","Type":"ContainerStarted","Data":"ddb792da3c8b169190fdd3a45f3cb8a6eb8e23e4773eaa7b37aff528bf831b71"} Jul 07 19:10:59.524901 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.524822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" event={"ID":"2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b","Type":"ContainerStarted","Data":"bcc7612670d142029caa7161c7a9986391c6aa35924b952941bec48f191f6a8b"} Jul 07 19:10:59.575673 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:10:59.575620 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-bbdd4f87-jfknr" podStartSLOduration=1.158566055 podStartE2EDuration="3.575600537s" totalImagesPullingTime="2.417034482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:59.574852491 +0000 UTC m=+165.510477877" watchObservedRunningTime="2026-07-07 19:10:59.575600537 +0000 UTC m=+165.511225896" Jul 07 19:11:00.526699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:00.526666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:00.536066 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:00.536032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e88c4bd6-8796-4ba1-8527-ad3677782c0f","Type":"ContainerStarted","Data":"75fef87131b6c4ffab9c2c5f63df3081e5f19732e6516230691c4855c3bd443c"} Jul 07 19:11:00.536251 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:00.536116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e88c4bd6-8796-4ba1-8527-ad3677782c0f","Type":"ContainerStarted","Data":"a0c666d9292d1880d723f2a75806bfa199da93f26390334338e0d703bf946996"} Jul 07 19:11:00.536251 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:00.536160 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e88c4bd6-8796-4ba1-8527-ad3677782c0f","Type":"ContainerStarted","Data":"6b49f3d12ca1a421ce9367798db994c19030f0d87c2dbe3e2e2603183f97ce43"} Jul 07 19:11:00.536251 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:00.536194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e88c4bd6-8796-4ba1-8527-ad3677782c0f","Type":"ContainerStarted","Data":"d6fc9b808122cd50e573666c5ca972b5e3dc1d22ef42dcff8c95de2b0a29d59f"} Jul 07 19:11:00.536251 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:00.536227 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e88c4bd6-8796-4ba1-8527-ad3677782c0f","Type":"ContainerStarted","Data":"ba4db56870710744c95ac73225994a51d232eede2acf14a940e37cc7fd6607a8"} Jul 07 19:11:00.536464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:00.536262 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e88c4bd6-8796-4ba1-8527-ad3677782c0f","Type":"ContainerStarted","Data":"2003bfbbe6404cd9dc134311a661826abb3daf501de8903eae48944819836767"} Jul 07 19:11:00.578809 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:00.578746 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.558837473 podStartE2EDuration="2.578726509s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.889036ms" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:00.57759906 +0000 UTC m=+166.513224436" watchObservedRunningTime="2026-07-07 19:11:00.578726509 +0000 UTC m=+166.514351868" Jul 07 19:11:00.618950 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:00.618918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:03.904436 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:03.904398 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:11:11.417456 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:11.417365 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:11.514728 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:11.514690 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:11.615109 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:11.615069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:35.438078 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:35.438038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:54.977523 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:54.977491 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:55.040082 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:55.040044 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:58.905037 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:58.905006 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:11:58.920961 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:58.920931 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:11:59.781643 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:11:59.781618 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:12:04.564916 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:04.564856 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:06.305779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:06.305744 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:06.408734 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:06.408697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:18.317079 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:18.317038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:18.416010 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:18.415976 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:24.828055 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:24.828020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:24.924883 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:24.924837 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:25.120942 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:25.120839 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:27.849022 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:27.848988 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:27.907615 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:27.907588 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:27.991019 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:27.990988 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:28.056170 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:28.056145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:28.593786 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:28.593754 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:28.687541 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:28.687506 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:38.778326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:38.778244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:38.878470 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:38.878430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:38.946265 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:12:38.946236 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:14.437998 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:14.437957 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 19:13:17.831270 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:17.831234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:19.071705 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:19.071666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:44.645343 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:44.645310 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:45.607828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:45.607792 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:48.265360 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:48.265326 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:48.367254 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:48.367220 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:49.816645 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:49.816613 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:49.918623 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:49.918594 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:49.953512 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:49.953482 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m6969"] Jul 07 19:13:49.956821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:49.956806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:13:49.959662 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:49.959636 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-97dcm\"" Jul 07 19:13:49.959819 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:49.959666 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 19:13:49.960626 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:49.960611 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 19:13:49.968548 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:49.968524 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m6969"] Jul 07 19:13:50.016923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.016893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be884bb7-360f-4080-a1e4-7bc4e5491554-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m6969\" (UID: \"be884bb7-360f-4080-a1e4-7bc4e5491554\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:13:50.016923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.016927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8624j\" (UniqueName: \"kubernetes.io/projected/be884bb7-360f-4080-a1e4-7bc4e5491554-kube-api-access-8624j\") pod \"cert-manager-webhook-6486ffd997-m6969\" (UID: \"be884bb7-360f-4080-a1e4-7bc4e5491554\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:13:50.117534 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.117449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be884bb7-360f-4080-a1e4-7bc4e5491554-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m6969\" (UID: \"be884bb7-360f-4080-a1e4-7bc4e5491554\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:13:50.117534 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.117488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8624j\" (UniqueName: \"kubernetes.io/projected/be884bb7-360f-4080-a1e4-7bc4e5491554-kube-api-access-8624j\") pod \"cert-manager-webhook-6486ffd997-m6969\" (UID: \"be884bb7-360f-4080-a1e4-7bc4e5491554\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:13:50.127188 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.127158 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8624j\" (UniqueName: \"kubernetes.io/projected/be884bb7-360f-4080-a1e4-7bc4e5491554-kube-api-access-8624j\") pod \"cert-manager-webhook-6486ffd997-m6969\" (UID: \"be884bb7-360f-4080-a1e4-7bc4e5491554\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:13:50.127304 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.127276 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be884bb7-360f-4080-a1e4-7bc4e5491554-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m6969\" (UID: \"be884bb7-360f-4080-a1e4-7bc4e5491554\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:13:50.274701 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.274668 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:13:50.316063 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.316023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:50.487961 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.485213 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m6969"] Jul 07 19:13:50.500603 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:13:50.500561 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe884bb7_360f_4080_a1e4_7bc4e5491554.slice/crio-898525b50b3fb9e1b02e98419d664d37e55f1b0fc0d6068fa397ddc87262dcd4 WatchSource:0}: Error finding container 898525b50b3fb9e1b02e98419d664d37e55f1b0fc0d6068fa397ddc87262dcd4: Status 404 returned error can't find the container with id 898525b50b3fb9e1b02e98419d664d37e55f1b0fc0d6068fa397ddc87262dcd4 Jul 07 19:13:50.502834 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.502814 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:13:50.503517 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:50.503498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:51.204269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:51.204231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" event={"ID":"be884bb7-360f-4080-a1e4-7bc4e5491554","Type":"ContainerStarted","Data":"898525b50b3fb9e1b02e98419d664d37e55f1b0fc0d6068fa397ddc87262dcd4"} Jul 07 19:13:52.211578 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:52.211540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:52.624071 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:52.624041 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:54.218115 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:54.218083 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" event={"ID":"be884bb7-360f-4080-a1e4-7bc4e5491554","Type":"ContainerStarted","Data":"f410b3edd957c82b3fe0e2a4c3a03bc1a6fb370e4fc3f7fbca826f044c915056"} Jul 07 19:13:54.218522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:54.218221 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:13:54.237690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:54.237636 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" podStartSLOduration=2.40138379 podStartE2EDuration="5.237622165s" totalImagesPullingTime="2.836238375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:13:54.236603745 +0000 UTC m=+340.172229109" watchObservedRunningTime="2026-07-07 19:13:54.237622165 +0000 UTC m=+340.173247524" Jul 07 19:13:55.025985 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:55.025950 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:55.090813 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:55.090778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:56.787212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.787160 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7"] Jul 07 19:13:56.790405 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.790387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" Jul 07 19:13:56.793934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.793913 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5wqb7\"" Jul 07 19:13:56.801926 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.801903 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7"] Jul 07 19:13:56.878542 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.878518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xxj8\" (UniqueName: \"kubernetes.io/projected/afab3396-0178-4be8-8382-b0f2e6fa7c58-kube-api-access-8xxj8\") pod \"cert-manager-cainjector-559cdd8c87-k9zm7\" (UID: \"afab3396-0178-4be8-8382-b0f2e6fa7c58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" Jul 07 19:13:56.878678 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.878588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afab3396-0178-4be8-8382-b0f2e6fa7c58-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k9zm7\" (UID: \"afab3396-0178-4be8-8382-b0f2e6fa7c58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" Jul 07 19:13:56.979171 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.979141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afab3396-0178-4be8-8382-b0f2e6fa7c58-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k9zm7\" (UID: \"afab3396-0178-4be8-8382-b0f2e6fa7c58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" Jul 07 19:13:56.979310 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.979190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxj8\" (UniqueName: \"kubernetes.io/projected/afab3396-0178-4be8-8382-b0f2e6fa7c58-kube-api-access-8xxj8\") pod \"cert-manager-cainjector-559cdd8c87-k9zm7\" (UID: \"afab3396-0178-4be8-8382-b0f2e6fa7c58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" Jul 07 19:13:56.988010 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.987988 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afab3396-0178-4be8-8382-b0f2e6fa7c58-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k9zm7\" (UID: \"afab3396-0178-4be8-8382-b0f2e6fa7c58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" Jul 07 19:13:56.988275 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:56.988253 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxj8\" (UniqueName: \"kubernetes.io/projected/afab3396-0178-4be8-8382-b0f2e6fa7c58-kube-api-access-8xxj8\") pod \"cert-manager-cainjector-559cdd8c87-k9zm7\" (UID: \"afab3396-0178-4be8-8382-b0f2e6fa7c58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" Jul 07 19:13:57.099059 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:57.099029 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" Jul 07 19:13:57.226948 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:57.226855 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7"] Jul 07 19:13:57.229958 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:13:57.229928 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafab3396_0178_4be8_8382_b0f2e6fa7c58.slice/crio-d2d6f464c22431d61c16a3bbaa66c709bb5d795b573efc6d9ae68935bd5d27d5 WatchSource:0}: Error finding container d2d6f464c22431d61c16a3bbaa66c709bb5d795b573efc6d9ae68935bd5d27d5: Status 404 returned error can't find the container with id d2d6f464c22431d61c16a3bbaa66c709bb5d795b573efc6d9ae68935bd5d27d5 Jul 07 19:13:58.064295 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:58.064254 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:58.153234 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:58.153203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:58.235938 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:58.235905 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" event={"ID":"afab3396-0178-4be8-8382-b0f2e6fa7c58","Type":"ContainerStarted","Data":"9b3927dd6c441dd7836b8ac5562955da33f31960190f35e4d05044e671775c18"} Jul 07 19:13:58.235938 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:58.235944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" event={"ID":"afab3396-0178-4be8-8382-b0f2e6fa7c58","Type":"ContainerStarted","Data":"d2d6f464c22431d61c16a3bbaa66c709bb5d795b573efc6d9ae68935bd5d27d5"} Jul 07 19:13:58.253083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:58.253033 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9zm7" podStartSLOduration=2.253015724 podStartE2EDuration="2.253015724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:13:58.252556917 +0000 UTC m=+344.188182277" watchObservedRunningTime="2026-07-07 19:13:58.253015724 +0000 UTC m=+344.188641083" Jul 07 19:13:59.438417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:13:59.438387 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:00.225129 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:00.225095 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m6969" Jul 07 19:14:01.450002 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:01.449968 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:01.573770 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:01.573735 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:06.721788 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:06.721704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:06.823113 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:06.823079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:08.404828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:08.404796 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:21.438398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:21.438363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:21.531124 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:21.531095 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:22.058719 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.058688 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:22.288793 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.288757 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:22.390253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.390168 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:22.479697 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.479669 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk"] Jul 07 19:14:22.483120 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.483105 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.486107 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.486083 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 19:14:22.486292 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.486094 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 19:14:22.486421 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.486094 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dfzj8\"" Jul 07 19:14:22.486646 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.486630 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 19:14:22.486696 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.486681 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 19:14:22.502003 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.501974 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk"] Jul 07 19:14:22.515541 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.515511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20d98418-bc55-46ad-a0a2-934ac44de34d-manifests\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.515677 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.515552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20d98418-bc55-46ad-a0a2-934ac44de34d-webhook-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.515677 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.515641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jmgq\" (UniqueName: \"kubernetes.io/projected/20d98418-bc55-46ad-a0a2-934ac44de34d-kube-api-access-4jmgq\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.515748 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.515674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/20d98418-bc55-46ad-a0a2-934ac44de34d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.515748 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.515720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20d98418-bc55-46ad-a0a2-934ac44de34d-tmp\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.596510 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.596477 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:22.616306 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.616279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmgq\" (UniqueName: \"kubernetes.io/projected/20d98418-bc55-46ad-a0a2-934ac44de34d-kube-api-access-4jmgq\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.616413 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.616310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/20d98418-bc55-46ad-a0a2-934ac44de34d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.616413 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.616342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20d98418-bc55-46ad-a0a2-934ac44de34d-tmp\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.616413 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.616385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20d98418-bc55-46ad-a0a2-934ac44de34d-manifests\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.616538 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.616414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20d98418-bc55-46ad-a0a2-934ac44de34d-webhook-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.616946 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.616925 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20d98418-bc55-46ad-a0a2-934ac44de34d-manifests\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.618788 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.618764 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20d98418-bc55-46ad-a0a2-934ac44de34d-tmp\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.619008 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.618991 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20d98418-bc55-46ad-a0a2-934ac44de34d-webhook-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.619056 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.619021 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/20d98418-bc55-46ad-a0a2-934ac44de34d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.625656 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.625637 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmgq\" (UniqueName: \"kubernetes.io/projected/20d98418-bc55-46ad-a0a2-934ac44de34d-kube-api-access-4jmgq\") pod \"opendatahub-operator-controller-manager-8689849645-cjrtk\" (UID: \"20d98418-bc55-46ad-a0a2-934ac44de34d\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.700105 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.700030 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:22.793359 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.793330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:22.819262 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.819234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:22.942060 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.942034 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk"] Jul 07 19:14:22.944856 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:14:22.944833 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d98418_bc55_46ad_a0a2_934ac44de34d.slice/crio-9528e13c4e6695f0422b8e75e89a369fd547ae71692594c5804eae0c39918328 WatchSource:0}: Error finding container 9528e13c4e6695f0422b8e75e89a369fd547ae71692594c5804eae0c39918328: Status 404 returned error can't find the container with id 9528e13c4e6695f0422b8e75e89a369fd547ae71692594c5804eae0c39918328 Jul 07 19:14:22.963070 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:22.963018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:23.065998 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:23.065955 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:23.183620 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:23.183593 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:23.334493 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:23.334461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" event={"ID":"20d98418-bc55-46ad-a0a2-934ac44de34d","Type":"ContainerStarted","Data":"9528e13c4e6695f0422b8e75e89a369fd547ae71692594c5804eae0c39918328"} Jul 07 19:14:27.355626 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:27.355586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" event={"ID":"20d98418-bc55-46ad-a0a2-934ac44de34d","Type":"ContainerStarted","Data":"61b574989b41deeb9938d4805b814c89dbeae6934a174cc18b7bf66924345612"} Jul 07 19:14:28.361604 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:28.361565 2577 generic.go:356] "Generic (PLEG): container finished" podID="20d98418-bc55-46ad-a0a2-934ac44de34d" containerID="61b574989b41deeb9938d4805b814c89dbeae6934a174cc18b7bf66924345612" exitCode=0 Jul 07 19:14:28.361994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:28.361622 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" event={"ID":"20d98418-bc55-46ad-a0a2-934ac44de34d","Type":"ContainerDied","Data":"61b574989b41deeb9938d4805b814c89dbeae6934a174cc18b7bf66924345612"} Jul 07 19:14:29.367584 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:29.367547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" event={"ID":"20d98418-bc55-46ad-a0a2-934ac44de34d","Type":"ContainerStarted","Data":"03ee26e51884d8f258e15f3f5e5c5b3a1062c9cc120b10f98487ad543a8f30ea"} Jul 07 19:14:29.368059 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:29.367651 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:29.395999 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:29.395953 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" podStartSLOduration=2.349125139 podStartE2EDuration="7.395938411s" totalImagesPullingTime="3.970213661s" totalInitContainerRuntime="1.076599611s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:29.39408508 +0000 UTC m=+375.329710439" watchObservedRunningTime="2026-07-07 19:14:29.395938411 +0000 UTC m=+375.331563839" Jul 07 19:14:33.674233 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:33.674195 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:33.817814 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:33.817783 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:38.469683 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:38.469646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:38.568355 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:38.568315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:40.373586 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:40.373559 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cjrtk" Jul 07 19:14:42.488857 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:42.488822 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:42.615104 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:42.615062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:43.051748 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:43.051674 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:43.463261 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:43.463149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:43.709673 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:43.709614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:43.831054 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:43.831019 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:43.918077 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:43.918047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:44.650540 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:44.650488 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:44.835038 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:44.835008 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:44.941021 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:44.940930 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:45.251361 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:45.251332 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:46.322839 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:46.322797 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:50.883822 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:50.883768 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:50.980455 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:50.980421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:51.217223 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.217110 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:51.385631 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.385590 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:51.407832 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.407793 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr"] Jul 07 19:14:51.411146 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.411126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.414689 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.414665 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 19:14:51.414824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.414737 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 19:14:51.415013 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.414992 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 19:14:51.415379 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.415364 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kgb6k\"" Jul 07 19:14:51.434069 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.434047 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr"] Jul 07 19:14:51.488091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.488009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.488091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.488039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.488091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.488081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.488398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.488161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.488398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.488192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.488398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.488228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gpf\" (UniqueName: \"kubernetes.io/projected/f44ed616-49f7-49ef-8db4-ac4135c3379b-kube-api-access-b8gpf\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.488398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.488270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.488398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.488304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.488398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.488322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44ed616-49f7-49ef-8db4-ac4135c3379b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.515101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.515076 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:51.589422 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.589568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.589568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gpf\" (UniqueName: \"kubernetes.io/projected/f44ed616-49f7-49ef-8db4-ac4135c3379b-kube-api-access-b8gpf\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.589568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589501 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.589695 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.589695 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44ed616-49f7-49ef-8db4-ac4135c3379b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.589791 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.589791 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.589911 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.590004 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.589984 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.590123 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.590101 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.590206 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.590189 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.590253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.590229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.590670 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.590650 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44ed616-49f7-49ef-8db4-ac4135c3379b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.591859 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.591838 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.592171 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.592155 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.604859 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.604836 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gpf\" (UniqueName: \"kubernetes.io/projected/f44ed616-49f7-49ef-8db4-ac4135c3379b-kube-api-access-b8gpf\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.605060 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.605041 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44ed616-49f7-49ef-8db4-ac4135c3379b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4wh7sr\" (UID: \"f44ed616-49f7-49ef-8db4-ac4135c3379b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.722185 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.722149 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:51.865548 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:51.865519 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr"] Jul 07 19:14:51.869047 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:14:51.869019 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44ed616_49f7_49ef_8db4_ac4135c3379b.slice/crio-dae3790b90dbbc6d1672e9b440061bd3b6a7ddb89ebc336969422f7a158253e9 WatchSource:0}: Error finding container dae3790b90dbbc6d1672e9b440061bd3b6a7ddb89ebc336969422f7a158253e9: Status 404 returned error can't find the container with id dae3790b90dbbc6d1672e9b440061bd3b6a7ddb89ebc336969422f7a158253e9 Jul 07 19:14:52.471748 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:52.471709 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" event={"ID":"f44ed616-49f7-49ef-8db4-ac4135c3379b","Type":"ContainerStarted","Data":"dae3790b90dbbc6d1672e9b440061bd3b6a7ddb89ebc336969422f7a158253e9"} Jul 07 19:14:53.532173 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.532139 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76"] Jul 07 19:14:53.538824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.538803 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.545557 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.545521 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:14:53.545722 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.545581 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 19:14:53.545722 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.545584 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 19:14:53.545722 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.545584 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 19:14:53.545999 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.545968 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rm5xg\"" Jul 07 19:14:53.546128 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.546021 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 19:14:53.566588 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.566558 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76"] Jul 07 19:14:53.708884 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.708837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-cert\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.709075 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.708989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-metrics-cert\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.709075 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.709050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqrkp\" (UniqueName: \"kubernetes.io/projected/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-kube-api-access-kqrkp\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.709191 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.709087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-manager-config\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.810140 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.810064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-metrics-cert\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.810140 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.810128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqrkp\" (UniqueName: \"kubernetes.io/projected/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-kube-api-access-kqrkp\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.810357 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.810161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-manager-config\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.810357 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.810188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-cert\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.810927 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.810906 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-manager-config\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.813285 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.813262 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-metrics-cert\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.813407 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.813309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-cert\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.820682 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.820659 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqrkp\" (UniqueName: \"kubernetes.io/projected/54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1-kube-api-access-kqrkp\") pod \"lws-controller-manager-5d95ccd586-snz76\" (UID: \"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:53.849009 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:53.848990 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:54.476580 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:54.476529 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:54.476663 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:54.476642 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:54.476710 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:54.476700 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:14:54.549125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:54.549102 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76"] Jul 07 19:14:54.551186 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:14:54.551166 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ab0ba6_dbe8_4923_8bdb_974b4c1d62d1.slice/crio-61d67d979ab5bdeec7cfcd16ed9dffbdb859f903346d1f0201db05d01de9fdf6 WatchSource:0}: Error finding container 61d67d979ab5bdeec7cfcd16ed9dffbdb859f903346d1f0201db05d01de9fdf6: Status 404 returned error can't find the container with id 61d67d979ab5bdeec7cfcd16ed9dffbdb859f903346d1f0201db05d01de9fdf6 Jul 07 19:14:54.939470 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:54.939431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:55.105180 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.105138 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:55.112970 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.112931 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-67ccf"] Jul 07 19:14:55.116803 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.116778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:14:55.117343 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.117305 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9"] Jul 07 19:14:55.119496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.119472 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zs6p2\"" Jul 07 19:14:55.119496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.119491 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 07 19:14:55.121012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.120994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:14:55.125261 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.125242 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 07 19:14:55.130012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.129990 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wnbgq\"" Jul 07 19:14:55.132021 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.132001 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-67ccf"] Jul 07 19:14:55.142192 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.142169 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9"] Jul 07 19:14:55.223563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.223482 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/468aba6c-4436-470a-a5de-34c2e24325b0-cert\") pod \"kserve-controller-manager-d6f4b65b8-67ccf\" (UID: \"468aba6c-4436-470a-a5de-34c2e24325b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:14:55.223563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.223549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxbvc\" (UniqueName: \"kubernetes.io/projected/468aba6c-4436-470a-a5de-34c2e24325b0-kube-api-access-jxbvc\") pod \"kserve-controller-manager-d6f4b65b8-67ccf\" (UID: \"468aba6c-4436-470a-a5de-34c2e24325b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:14:55.223730 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.223577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d921ac91-a173-4000-9da8-26c6037abeaa-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tt9z9\" (UID: \"d921ac91-a173-4000-9da8-26c6037abeaa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:14:55.223730 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.223612 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxpxh\" (UniqueName: \"kubernetes.io/projected/d921ac91-a173-4000-9da8-26c6037abeaa-kube-api-access-mxpxh\") pod \"llmisvc-controller-manager-777c6bd7d5-tt9z9\" (UID: \"d921ac91-a173-4000-9da8-26c6037abeaa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:14:55.305574 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.305540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:55.325094 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.325062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/468aba6c-4436-470a-a5de-34c2e24325b0-cert\") pod \"kserve-controller-manager-d6f4b65b8-67ccf\" (UID: \"468aba6c-4436-470a-a5de-34c2e24325b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:14:55.325292 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.325192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxbvc\" (UniqueName: \"kubernetes.io/projected/468aba6c-4436-470a-a5de-34c2e24325b0-kube-api-access-jxbvc\") pod \"kserve-controller-manager-d6f4b65b8-67ccf\" (UID: \"468aba6c-4436-470a-a5de-34c2e24325b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:14:55.325292 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:14:55.325201 2577 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 07 19:14:55.325764 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.325246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d921ac91-a173-4000-9da8-26c6037abeaa-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tt9z9\" (UID: \"d921ac91-a173-4000-9da8-26c6037abeaa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:14:55.325764 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:14:55.325359 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/468aba6c-4436-470a-a5de-34c2e24325b0-cert podName:468aba6c-4436-470a-a5de-34c2e24325b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:14:55.825333247 +0000 UTC m=+401.760958594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/468aba6c-4436-470a-a5de-34c2e24325b0-cert") pod "kserve-controller-manager-d6f4b65b8-67ccf" (UID: "468aba6c-4436-470a-a5de-34c2e24325b0") : secret "kserve-webhook-server-cert" not found Jul 07 19:14:55.325764 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.325432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxpxh\" (UniqueName: \"kubernetes.io/projected/d921ac91-a173-4000-9da8-26c6037abeaa-kube-api-access-mxpxh\") pod \"llmisvc-controller-manager-777c6bd7d5-tt9z9\" (UID: \"d921ac91-a173-4000-9da8-26c6037abeaa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:14:55.329337 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.329315 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d921ac91-a173-4000-9da8-26c6037abeaa-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tt9z9\" (UID: \"d921ac91-a173-4000-9da8-26c6037abeaa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:14:55.338154 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.338128 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxpxh\" (UniqueName: \"kubernetes.io/projected/d921ac91-a173-4000-9da8-26c6037abeaa-kube-api-access-mxpxh\") pod \"llmisvc-controller-manager-777c6bd7d5-tt9z9\" (UID: \"d921ac91-a173-4000-9da8-26c6037abeaa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:14:55.344620 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.344588 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxbvc\" (UniqueName: \"kubernetes.io/projected/468aba6c-4436-470a-a5de-34c2e24325b0-kube-api-access-jxbvc\") pod \"kserve-controller-manager-d6f4b65b8-67ccf\" (UID: \"468aba6c-4436-470a-a5de-34c2e24325b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:14:55.437498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.437464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:14:55.493599 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.493520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" event={"ID":"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1","Type":"ContainerStarted","Data":"61d67d979ab5bdeec7cfcd16ed9dffbdb859f903346d1f0201db05d01de9fdf6"} Jul 07 19:14:55.497166 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.497135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" event={"ID":"f44ed616-49f7-49ef-8db4-ac4135c3379b","Type":"ContainerStarted","Data":"4a831d166f4b701301027be18ae510eceb8c4abd62741345ac9f3e8de3bf41b5"} Jul 07 19:14:55.555500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.553889 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" podStartSLOduration=1.948583552 podStartE2EDuration="4.553849717s" totalImagesPullingTime="2.605266165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:55.550918232 +0000 UTC m=+401.486543592" watchObservedRunningTime="2026-07-07 19:14:55.553849717 +0000 UTC m=+401.489475077" Jul 07 19:14:55.618627 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.618598 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9"] Jul 07 19:14:55.621222 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:14:55.621189 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd921ac91_a173_4000_9da8_26c6037abeaa.slice/crio-47b181c1eca5442549b82d5d462e83c52c2c4ad6bfb3e1a06cb14a9519d2887b WatchSource:0}: Error finding container 47b181c1eca5442549b82d5d462e83c52c2c4ad6bfb3e1a06cb14a9519d2887b: Status 404 returned error can't find the container with id 47b181c1eca5442549b82d5d462e83c52c2c4ad6bfb3e1a06cb14a9519d2887b Jul 07 19:14:55.704981 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.704942 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:55.723253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.723222 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:55.730096 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.730023 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:55.831966 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.831380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/468aba6c-4436-470a-a5de-34c2e24325b0-cert\") pod \"kserve-controller-manager-d6f4b65b8-67ccf\" (UID: \"468aba6c-4436-470a-a5de-34c2e24325b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:14:55.841456 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:55.839587 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/468aba6c-4436-470a-a5de-34c2e24325b0-cert\") pod \"kserve-controller-manager-d6f4b65b8-67ccf\" (UID: \"468aba6c-4436-470a-a5de-34c2e24325b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:14:56.031175 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:56.031145 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:14:56.413356 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:56.413328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:56.504031 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:56.503996 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" event={"ID":"d921ac91-a173-4000-9da8-26c6037abeaa","Type":"ContainerStarted","Data":"47b181c1eca5442549b82d5d462e83c52c2c4ad6bfb3e1a06cb14a9519d2887b"} Jul 07 19:14:56.504297 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:56.504272 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:56.505501 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:56.505473 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4wh7sr" Jul 07 19:14:56.530891 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:56.530854 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:57.172502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:57.172481 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-67ccf"] Jul 07 19:14:57.195565 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:14:57.195534 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468aba6c_4436_470a_a5de_34c2e24325b0.slice/crio-6d4a382dcdc69d5f5fd89ad2a2baaff37dba4f8832e3aa9c96181143c11417fc WatchSource:0}: Error finding container 6d4a382dcdc69d5f5fd89ad2a2baaff37dba4f8832e3aa9c96181143c11417fc: Status 404 returned error can't find the container with id 6d4a382dcdc69d5f5fd89ad2a2baaff37dba4f8832e3aa9c96181143c11417fc Jul 07 19:14:57.510426 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:57.510388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" event={"ID":"468aba6c-4436-470a-a5de-34c2e24325b0","Type":"ContainerStarted","Data":"6d4a382dcdc69d5f5fd89ad2a2baaff37dba4f8832e3aa9c96181143c11417fc"} Jul 07 19:14:58.517349 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:58.517309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" event={"ID":"54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1","Type":"ContainerStarted","Data":"141d8062c23a793a6c6ce65635bdbdeac5a485798ea70115aab0522dbcb665e6"} Jul 07 19:14:58.517857 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:58.517684 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:14:58.555827 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:14:58.555774 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" podStartSLOduration=2.913941187 podStartE2EDuration="5.555758738s" totalImagesPullingTime="2.641817551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:58.554004986 +0000 UTC m=+404.489630347" watchObservedRunningTime="2026-07-07 19:14:58.555758738 +0000 UTC m=+404.491384098" Jul 07 19:15:01.533031 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:01.532993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" event={"ID":"d921ac91-a173-4000-9da8-26c6037abeaa","Type":"ContainerStarted","Data":"130d67da2c9e6433ad40570912cade7f60d4765fadbef717776e012cb86daa6f"} Jul 07 19:15:01.533468 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:01.533072 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:15:01.534397 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:01.534380 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" event={"ID":"468aba6c-4436-470a-a5de-34c2e24325b0","Type":"ContainerStarted","Data":"7af6ae90323331f304a0997381a440a8b35fee7cd255609ce2f23d21705153b5"} Jul 07 19:15:01.534478 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:01.534418 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:15:01.554809 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:01.554757 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" podStartSLOduration=1.472059757 podStartE2EDuration="6.55474498s" totalImagesPullingTime="5.082685223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:15:01.552814156 +0000 UTC m=+407.488439509" watchObservedRunningTime="2026-07-07 19:15:01.55474498 +0000 UTC m=+407.490370340" Jul 07 19:15:01.575060 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:01.575021 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" podStartSLOduration=3.065711053 podStartE2EDuration="6.57500815s" totalImagesPullingTime="3.509297097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:15:01.574302988 +0000 UTC m=+407.509928339" watchObservedRunningTime="2026-07-07 19:15:01.57500815 +0000 UTC m=+407.510633509" Jul 07 19:15:09.525242 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:09.525214 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-snz76" Jul 07 19:15:32.540580 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:32.540552 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tt9z9" Jul 07 19:15:32.543550 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:32.543527 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-67ccf" Jul 07 19:15:43.944767 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:43.944696 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:44.041857 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:44.041825 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:44.127916 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:44.127884 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:51.390451 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:51.390414 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:52.137617 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:52.137588 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:52.207380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:52.207340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:54.084223 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:15:54.084185 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:11.427562 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:11.427530 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:12.868099 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:12.868063 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:16.460936 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:16.460900 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:16.563003 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:16.562964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:20.727718 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.727688 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs"] Jul 07 19:16:20.730064 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.730045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:20.734194 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.734171 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hgr87\"" Jul 07 19:16:20.734497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.734482 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 19:16:20.734564 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.734508 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 19:16:20.748202 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.748173 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs"] Jul 07 19:16:20.790578 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.790546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnsdp\" (UniqueName: \"kubernetes.io/projected/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-kube-api-access-fnsdp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" (UID: \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:20.790756 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.790637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" (UID: \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:20.891081 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.891051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" (UID: \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:20.891256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.891109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnsdp\" (UniqueName: \"kubernetes.io/projected/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-kube-api-access-fnsdp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" (UID: \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:20.891421 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.891404 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" (UID: \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:20.904816 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:20.904783 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnsdp\" (UniqueName: \"kubernetes.io/projected/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-kube-api-access-fnsdp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" (UID: \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:21.040408 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:21.040374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:21.048741 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:21.048715 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:21.177634 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:21.177605 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs"] Jul 07 19:16:21.180763 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:16:21.180735 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb10e1ef_3786_4e63_b2ed_9ac822dd6e01.slice/crio-a70c45207054693288a9388d0e1fd4237c3a7d6e051660f9441a046e8de85588 WatchSource:0}: Error finding container a70c45207054693288a9388d0e1fd4237c3a7d6e051660f9441a046e8de85588: Status 404 returned error can't find the container with id a70c45207054693288a9388d0e1fd4237c3a7d6e051660f9441a046e8de85588 Jul 07 19:16:21.439108 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:21.439039 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:21.897442 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:21.897391 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" event={"ID":"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01","Type":"ContainerStarted","Data":"a70c45207054693288a9388d0e1fd4237c3a7d6e051660f9441a046e8de85588"} Jul 07 19:16:22.934810 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:22.934782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:24.454195 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:24.454162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:25.334981 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:25.334945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:25.439284 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:25.439254 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:26.703898 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:26.703834 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk"] Jul 07 19:16:26.706524 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:26.706500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" Jul 07 19:16:26.709372 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:26.709353 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 07 19:16:26.709471 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:26.709382 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jfmfm\"" Jul 07 19:16:26.722372 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:26.722296 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk"] Jul 07 19:16:26.848995 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:26.848962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwpl\" (UniqueName: \"kubernetes.io/projected/32837b00-0905-4f99-a9f3-8f2f61438693-kube-api-access-vhwpl\") pod \"dns-operator-controller-manager-864dd78c4c-z6vwk\" (UID: \"32837b00-0905-4f99-a9f3-8f2f61438693\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" Jul 07 19:16:26.950026 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:26.949987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwpl\" (UniqueName: \"kubernetes.io/projected/32837b00-0905-4f99-a9f3-8f2f61438693-kube-api-access-vhwpl\") pod \"dns-operator-controller-manager-864dd78c4c-z6vwk\" (UID: \"32837b00-0905-4f99-a9f3-8f2f61438693\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" Jul 07 19:16:26.959554 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:26.959489 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwpl\" (UniqueName: \"kubernetes.io/projected/32837b00-0905-4f99-a9f3-8f2f61438693-kube-api-access-vhwpl\") pod \"dns-operator-controller-manager-864dd78c4c-z6vwk\" (UID: \"32837b00-0905-4f99-a9f3-8f2f61438693\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" Jul 07 19:16:27.017308 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:27.017280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" Jul 07 19:16:27.169013 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:27.168987 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk"] Jul 07 19:16:27.301898 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:27.301839 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:28.332939 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:16:28.332899 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32837b00_0905_4f99_a9f3_8f2f61438693.slice/crio-49bd6dd5812b5df36a634f6cefca42af72145bd094aca00a4c126e4ca39acd45 WatchSource:0}: Error finding container 49bd6dd5812b5df36a634f6cefca42af72145bd094aca00a4c126e4ca39acd45: Status 404 returned error can't find the container with id 49bd6dd5812b5df36a634f6cefca42af72145bd094aca00a4c126e4ca39acd45 Jul 07 19:16:28.660142 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:28.660054 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:28.768825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:28.768796 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:28.938943 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:28.938843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" event={"ID":"32837b00-0905-4f99-a9f3-8f2f61438693","Type":"ContainerStarted","Data":"49bd6dd5812b5df36a634f6cefca42af72145bd094aca00a4c126e4ca39acd45"} Jul 07 19:16:28.940447 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:28.940423 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" event={"ID":"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01","Type":"ContainerStarted","Data":"414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680"} Jul 07 19:16:28.940572 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:28.940556 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:28.966800 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:28.966749 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" podStartSLOduration=1.781075495 podStartE2EDuration="8.96672801s" totalImagesPullingTime="7.185652515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:28.966444978 +0000 UTC m=+494.902070340" watchObservedRunningTime="2026-07-07 19:16:28.96672801 +0000 UTC m=+494.902353370" Jul 07 19:16:30.952429 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:30.952391 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" event={"ID":"32837b00-0905-4f99-a9f3-8f2f61438693","Type":"ContainerStarted","Data":"2121f8b3ea92a7a7df75fe10b6b315c5fcd367cecb8a9466ba0da2cf2765e00c"} Jul 07 19:16:30.952932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:30.952529 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" Jul 07 19:16:30.994444 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:30.993756 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" podStartSLOduration=2.855509842 podStartE2EDuration="4.993732668s" totalImagesPullingTime="2.138222826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:30.992276788 +0000 UTC m=+496.927902151" watchObservedRunningTime="2026-07-07 19:16:30.993732668 +0000 UTC m=+496.929358029" Jul 07 19:16:31.468609 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:31.468573 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:31.709297 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:31.709270 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:31.863436 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:31.863407 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:31.963312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:31.963284 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:39.947778 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:39.947744 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:40.784968 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:40.784935 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:40.957979 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:40.957947 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:41.763653 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.763617 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs"] Jul 07 19:16:41.763851 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.763824 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" containerName="manager" containerID="cri-o://414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680" gracePeriod=2 Jul 07 19:16:41.772043 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.772017 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs"] Jul 07 19:16:41.807951 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.807926 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk"] Jul 07 19:16:41.808302 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.808289 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" containerName="manager" Jul 07 19:16:41.808344 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.808306 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" containerName="manager" Jul 07 19:16:41.808385 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.808376 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" containerName="manager" Jul 07 19:16:41.810981 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.810959 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:41.831246 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.831221 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk"] Jul 07 19:16:41.861401 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:16:41.861354 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" is forbidden: User \"system:node:ip-10-0-132-59.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-59.ec2.internal' and this object" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:41.893703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.893653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrqq\" (UniqueName: \"kubernetes.io/projected/53b30203-277b-48d3-844b-60030e8e0525-kube-api-access-4jrqq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bf5gk\" (UID: \"53b30203-277b-48d3-844b-60030e8e0525\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:41.893703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.893701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53b30203-277b-48d3-844b-60030e8e0525-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bf5gk\" (UID: \"53b30203-277b-48d3-844b-60030e8e0525\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:41.959604 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.959575 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z6vwk" Jul 07 19:16:41.988186 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:16:41.988155 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" is forbidden: User \"system:node:ip-10-0-132-59.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-59.ec2.internal' and this object" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:41.994890 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.994850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrqq\" (UniqueName: \"kubernetes.io/projected/53b30203-277b-48d3-844b-60030e8e0525-kube-api-access-4jrqq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bf5gk\" (UID: \"53b30203-277b-48d3-844b-60030e8e0525\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:41.995038 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.994913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53b30203-277b-48d3-844b-60030e8e0525-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bf5gk\" (UID: \"53b30203-277b-48d3-844b-60030e8e0525\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:41.995439 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:41.995417 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53b30203-277b-48d3-844b-60030e8e0525-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bf5gk\" (UID: \"53b30203-277b-48d3-844b-60030e8e0525\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:42.006374 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.006352 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:42.008744 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:16:42.008718 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" is forbidden: User \"system:node:ip-10-0-132-59.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-59.ec2.internal' and this object" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:42.008914 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.008894 2577 generic.go:356] "Generic (PLEG): container finished" podID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" containerID="414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680" exitCode=0 Jul 07 19:16:42.008993 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.008930 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:42.009057 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.008992 2577 scope.go:117] "RemoveContainer" containerID="414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680" Jul 07 19:16:42.011620 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:16:42.011595 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" is forbidden: User \"system:node:ip-10-0-132-59.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-59.ec2.internal' and this object" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:42.017975 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.017959 2577 scope.go:117] "RemoveContainer" containerID="414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680" Jul 07 19:16:42.018265 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:16:42.018245 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680\": container with ID starting with 414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680 not found: ID does not exist" containerID="414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680" Jul 07 19:16:42.018338 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.018277 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680"} err="failed to get container status \"414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680\": rpc error: code = NotFound desc = could not find container \"414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680\": container with ID starting with 414b6810a169a48589dfda83d2fc24ddf14b823f32083cdbdb79cb09fec91680 not found: ID does not exist" Jul 07 19:16:42.019617 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.019590 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrqq\" (UniqueName: \"kubernetes.io/projected/53b30203-277b-48d3-844b-60030e8e0525-kube-api-access-4jrqq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bf5gk\" (UID: \"53b30203-277b-48d3-844b-60030e8e0525\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:42.096209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.096182 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-extensions-socket-volume\") pod \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\" (UID: \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\") " Jul 07 19:16:42.096381 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.096285 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnsdp\" (UniqueName: \"kubernetes.io/projected/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-kube-api-access-fnsdp\") pod \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\" (UID: \"eb10e1ef-3786-4e63-b2ed-9ac822dd6e01\") " Jul 07 19:16:42.096622 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.096598 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" (UID: "eb10e1ef-3786-4e63-b2ed-9ac822dd6e01"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:42.098524 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.098504 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-kube-api-access-fnsdp" (OuterVolumeSpecName: "kube-api-access-fnsdp") pod "eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" (UID: "eb10e1ef-3786-4e63-b2ed-9ac822dd6e01"). InnerVolumeSpecName "kube-api-access-fnsdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:16:42.140732 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.140703 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:42.197907 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.197866 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnsdp\" (UniqueName: \"kubernetes.io/projected/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-kube-api-access-fnsdp\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:16:42.197907 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.197908 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01-extensions-socket-volume\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:16:42.287050 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.287027 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk"] Jul 07 19:16:42.289717 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:16:42.289691 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b30203_277b_48d3_844b_60030e8e0525.slice/crio-3986e3bcee4cfdb8b22a13632682cbd4ed09bdd0793cf4582e43bd64a502c61d WatchSource:0}: Error finding container 3986e3bcee4cfdb8b22a13632682cbd4ed09bdd0793cf4582e43bd64a502c61d: Status 404 returned error can't find the container with id 3986e3bcee4cfdb8b22a13632682cbd4ed09bdd0793cf4582e43bd64a502c61d Jul 07 19:16:42.335921 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:16:42.335890 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zffbs\" is forbidden: User \"system:node:ip-10-0-132-59.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-59.ec2.internal' and this object" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zffbs" Jul 07 19:16:42.604259 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:42.604190 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb10e1ef-3786-4e63-b2ed-9ac822dd6e01" path="/var/lib/kubelet/pods/eb10e1ef-3786-4e63-b2ed-9ac822dd6e01/volumes" Jul 07 19:16:43.016106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.016070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" event={"ID":"53b30203-277b-48d3-844b-60030e8e0525","Type":"ContainerStarted","Data":"afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69"} Jul 07 19:16:43.016495 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.016126 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" event={"ID":"53b30203-277b-48d3-844b-60030e8e0525","Type":"ContainerStarted","Data":"3986e3bcee4cfdb8b22a13632682cbd4ed09bdd0793cf4582e43bd64a502c61d"} Jul 07 19:16:43.016495 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.016164 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:43.044511 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.044466 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" podStartSLOduration=2.044449453 podStartE2EDuration="2.044449453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:43.043072557 +0000 UTC m=+508.978697917" watchObservedRunningTime="2026-07-07 19:16:43.044449453 +0000 UTC m=+508.980074813" Jul 07 19:16:43.192389 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.192358 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc"] Jul 07 19:16:43.196126 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.196099 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:43.205201 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.205176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vdm\" (UniqueName: \"kubernetes.io/projected/28159cac-8ea7-4248-8de5-90290ec0e51e-kube-api-access-n4vdm\") pod \"kuadrant-operator-controller-manager-86878c769f-mqqmc\" (UID: \"28159cac-8ea7-4248-8de5-90290ec0e51e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:43.205334 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.205267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28159cac-8ea7-4248-8de5-90290ec0e51e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mqqmc\" (UID: \"28159cac-8ea7-4248-8de5-90290ec0e51e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:43.215267 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.215241 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc"] Jul 07 19:16:43.242073 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.242050 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:43.306030 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.305960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vdm\" (UniqueName: \"kubernetes.io/projected/28159cac-8ea7-4248-8de5-90290ec0e51e-kube-api-access-n4vdm\") pod \"kuadrant-operator-controller-manager-86878c769f-mqqmc\" (UID: \"28159cac-8ea7-4248-8de5-90290ec0e51e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:43.306159 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.306034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28159cac-8ea7-4248-8de5-90290ec0e51e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mqqmc\" (UID: \"28159cac-8ea7-4248-8de5-90290ec0e51e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:43.306366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.306351 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28159cac-8ea7-4248-8de5-90290ec0e51e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mqqmc\" (UID: \"28159cac-8ea7-4248-8de5-90290ec0e51e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:43.323824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.323802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vdm\" (UniqueName: \"kubernetes.io/projected/28159cac-8ea7-4248-8de5-90290ec0e51e-kube-api-access-n4vdm\") pod \"kuadrant-operator-controller-manager-86878c769f-mqqmc\" (UID: \"28159cac-8ea7-4248-8de5-90290ec0e51e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:43.352569 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.352542 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:43.507068 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.507035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:43.666077 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:43.666054 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc"] Jul 07 19:16:43.668497 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:16:43.668470 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28159cac_8ea7_4248_8de5_90290ec0e51e.slice/crio-01243a89d24f336249349c3d69d2592406f5de4331cc86c7d2fb4ce91db74b7a WatchSource:0}: Error finding container 01243a89d24f336249349c3d69d2592406f5de4331cc86c7d2fb4ce91db74b7a: Status 404 returned error can't find the container with id 01243a89d24f336249349c3d69d2592406f5de4331cc86c7d2fb4ce91db74b7a Jul 07 19:16:44.022588 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:44.022559 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" event={"ID":"28159cac-8ea7-4248-8de5-90290ec0e51e","Type":"ContainerStarted","Data":"1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a"} Jul 07 19:16:44.022992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:44.022638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" event={"ID":"28159cac-8ea7-4248-8de5-90290ec0e51e","Type":"ContainerStarted","Data":"01243a89d24f336249349c3d69d2592406f5de4331cc86c7d2fb4ce91db74b7a"} Jul 07 19:16:44.022992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:44.022665 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:44.054964 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:44.054910 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" podStartSLOduration=1.054895559 podStartE2EDuration="1.054895559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:44.052196391 +0000 UTC m=+509.987821751" watchObservedRunningTime="2026-07-07 19:16:44.054895559 +0000 UTC m=+509.990520955" Jul 07 19:16:47.503666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:47.503633 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:47.609254 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:47.609223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:49.672507 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:49.672470 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:49.773747 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:49.773719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:54.024989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:54.024963 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:55.029465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.029436 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:16:55.085782 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.085739 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk"] Jul 07 19:16:55.086001 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.085920 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" podUID="53b30203-277b-48d3-844b-60030e8e0525" containerName="manager" containerID="cri-o://afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69" gracePeriod=10 Jul 07 19:16:55.341548 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.341522 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:55.411013 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.410979 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53b30203-277b-48d3-844b-60030e8e0525-extensions-socket-volume\") pod \"53b30203-277b-48d3-844b-60030e8e0525\" (UID: \"53b30203-277b-48d3-844b-60030e8e0525\") " Jul 07 19:16:55.411179 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.411121 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jrqq\" (UniqueName: \"kubernetes.io/projected/53b30203-277b-48d3-844b-60030e8e0525-kube-api-access-4jrqq\") pod \"53b30203-277b-48d3-844b-60030e8e0525\" (UID: \"53b30203-277b-48d3-844b-60030e8e0525\") " Jul 07 19:16:55.411348 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.411325 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b30203-277b-48d3-844b-60030e8e0525-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "53b30203-277b-48d3-844b-60030e8e0525" (UID: "53b30203-277b-48d3-844b-60030e8e0525"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:55.413162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.413137 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b30203-277b-48d3-844b-60030e8e0525-kube-api-access-4jrqq" (OuterVolumeSpecName: "kube-api-access-4jrqq") pod "53b30203-277b-48d3-844b-60030e8e0525" (UID: "53b30203-277b-48d3-844b-60030e8e0525"). InnerVolumeSpecName "kube-api-access-4jrqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:16:55.512380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.512338 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53b30203-277b-48d3-844b-60030e8e0525-extensions-socket-volume\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:16:55.512380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:55.512372 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrqq\" (UniqueName: \"kubernetes.io/projected/53b30203-277b-48d3-844b-60030e8e0525-kube-api-access-4jrqq\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:16:56.080659 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.080625 2577 generic.go:356] "Generic (PLEG): container finished" podID="53b30203-277b-48d3-844b-60030e8e0525" containerID="afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69" exitCode=0 Jul 07 19:16:56.081094 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.080666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" event={"ID":"53b30203-277b-48d3-844b-60030e8e0525","Type":"ContainerDied","Data":"afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69"} Jul 07 19:16:56.081094 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.080687 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" Jul 07 19:16:56.081094 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.080706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk" event={"ID":"53b30203-277b-48d3-844b-60030e8e0525","Type":"ContainerDied","Data":"3986e3bcee4cfdb8b22a13632682cbd4ed09bdd0793cf4582e43bd64a502c61d"} Jul 07 19:16:56.081094 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.080727 2577 scope.go:117] "RemoveContainer" containerID="afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69" Jul 07 19:16:56.089608 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.089398 2577 scope.go:117] "RemoveContainer" containerID="afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69" Jul 07 19:16:56.089697 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:16:56.089675 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69\": container with ID starting with afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69 not found: ID does not exist" containerID="afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69" Jul 07 19:16:56.089738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.089706 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69"} err="failed to get container status \"afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69\": rpc error: code = NotFound desc = could not find container \"afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69\": container with ID starting with afe57ab63812012a88cb4649b19000bbefe83d892453a387f3c49759624c7b69 not found: ID does not exist" Jul 07 19:16:56.105839 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.105811 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk"] Jul 07 19:16:56.114051 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.114030 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bf5gk"] Jul 07 19:16:56.603432 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:56.603392 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53b30203-277b-48d3-844b-60030e8e0525" path="/var/lib/kubelet/pods/53b30203-277b-48d3-844b-60030e8e0525/volumes" Jul 07 19:16:58.540693 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:58.540661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:58.641366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:16:58.641334 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:00.777461 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:00.777429 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:00.877616 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:00.877582 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:00.979245 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:00.979210 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:01.078020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:01.077938 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:04.905433 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:04.905348 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:05.340051 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:05.340014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:05.545677 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:05.545644 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:08.414377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:08.414344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:08.517590 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:08.517556 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:11.443063 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.443022 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7"] Jul 07 19:17:11.443573 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.443554 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b30203-277b-48d3-844b-60030e8e0525" containerName="manager" Jul 07 19:17:11.443656 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.443576 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b30203-277b-48d3-844b-60030e8e0525" containerName="manager" Jul 07 19:17:11.443918 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.443695 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b30203-277b-48d3-844b-60030e8e0525" containerName="manager" Jul 07 19:17:11.446629 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.446608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.449284 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.449265 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-px6f8\"" Jul 07 19:17:11.462848 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.462825 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7"] Jul 07 19:17:11.490781 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.490755 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:11.559108 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.559074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.559297 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.559128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93df3e10-d156-40e0-8f81-367099052b5f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.559297 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.559168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkcmg\" (UniqueName: \"kubernetes.io/projected/93df3e10-d156-40e0-8f81-367099052b5f-kube-api-access-tkcmg\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.559297 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.559275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.559464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.559313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.559464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.559340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.559576 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.559460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.559576 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.559501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93df3e10-d156-40e0-8f81-367099052b5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.559672 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.559579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93df3e10-d156-40e0-8f81-367099052b5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.660602 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.660571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkcmg\" (UniqueName: \"kubernetes.io/projected/93df3e10-d156-40e0-8f81-367099052b5f-kube-api-access-tkcmg\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.660750 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.660611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.660750 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.660627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.660750 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.660669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.660750 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.660725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.660750 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.660745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93df3e10-d156-40e0-8f81-367099052b5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.661014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.660792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93df3e10-d156-40e0-8f81-367099052b5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.661014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.660920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.661014 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.660968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93df3e10-d156-40e0-8f81-367099052b5f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.661190 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.661169 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.661251 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.661229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.661378 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.661345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.661558 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.661541 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.661634 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.661614 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93df3e10-d156-40e0-8f81-367099052b5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.663579 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.663555 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93df3e10-d156-40e0-8f81-367099052b5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.663731 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.663715 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93df3e10-d156-40e0-8f81-367099052b5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.671123 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.671100 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93df3e10-d156-40e0-8f81-367099052b5f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.671228 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.671165 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkcmg\" (UniqueName: \"kubernetes.io/projected/93df3e10-d156-40e0-8f81-367099052b5f-kube-api-access-tkcmg\") pod \"maas-default-gateway-openshift-default-56654dcf7-d6nj7\" (UID: \"93df3e10-d156-40e0-8f81-367099052b5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.757293 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.757249 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:11.818192 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.818158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:11.912637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.912603 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7"] Jul 07 19:17:11.914346 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:17:11.914322 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93df3e10_d156_40e0_8f81_367099052b5f.slice/crio-433788f586583e0a77d48b10e469f7f356752c97342527f76c2e05137f2df651 WatchSource:0}: Error finding container 433788f586583e0a77d48b10e469f7f356752c97342527f76c2e05137f2df651: Status 404 returned error can't find the container with id 433788f586583e0a77d48b10e469f7f356752c97342527f76c2e05137f2df651 Jul 07 19:17:11.916499 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.916467 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:11.916563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.916536 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:11.916605 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:11.916566 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:17:12.162222 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:12.162135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" event={"ID":"93df3e10-d156-40e0-8f81-367099052b5f","Type":"ContainerStarted","Data":"059064c939f107614ff98d0d34d2e40081a4dfffeb750b314acd85f2eff69489"} Jul 07 19:17:12.162222 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:12.162185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" event={"ID":"93df3e10-d156-40e0-8f81-367099052b5f","Type":"ContainerStarted","Data":"433788f586583e0a77d48b10e469f7f356752c97342527f76c2e05137f2df651"} Jul 07 19:17:12.187443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:12.187398 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" podStartSLOduration=1.18738229 podStartE2EDuration="1.18738229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:12.185967064 +0000 UTC m=+538.121592425" watchObservedRunningTime="2026-07-07 19:17:12.18738229 +0000 UTC m=+538.123007649" Jul 07 19:17:12.758431 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:12.758396 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:12.763214 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:12.763189 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:13.167819 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.167735 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:13.168588 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.168567 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:17:13.390659 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.390617 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2fv9"] Jul 07 19:17:13.394982 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.394959 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:13.396552 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.396523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:13.398344 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.398320 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6d87x\"" Jul 07 19:17:13.398765 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.398716 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 19:17:13.419689 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.419633 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2fv9"] Jul 07 19:17:13.478281 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.478252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80d78dd3-0bfd-477d-96b2-435eff5b83fd-config-file\") pod \"limitador-limitador-64c94fd64f-d2fv9\" (UID: \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:13.478507 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.478375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kpw4\" (UniqueName: \"kubernetes.io/projected/80d78dd3-0bfd-477d-96b2-435eff5b83fd-kube-api-access-8kpw4\") pod \"limitador-limitador-64c94fd64f-d2fv9\" (UID: \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:13.496647 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.496618 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2fv9"] Jul 07 19:17:13.504474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.504448 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:13.579182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.579151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80d78dd3-0bfd-477d-96b2-435eff5b83fd-config-file\") pod \"limitador-limitador-64c94fd64f-d2fv9\" (UID: \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:13.579350 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.579240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kpw4\" (UniqueName: \"kubernetes.io/projected/80d78dd3-0bfd-477d-96b2-435eff5b83fd-kube-api-access-8kpw4\") pod \"limitador-limitador-64c94fd64f-d2fv9\" (UID: \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:13.579806 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.579787 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80d78dd3-0bfd-477d-96b2-435eff5b83fd-config-file\") pod \"limitador-limitador-64c94fd64f-d2fv9\" (UID: \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:13.594385 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.594357 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kpw4\" (UniqueName: \"kubernetes.io/projected/80d78dd3-0bfd-477d-96b2-435eff5b83fd-kube-api-access-8kpw4\") pod \"limitador-limitador-64c94fd64f-d2fv9\" (UID: \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:13.706239 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.706145 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:13.847464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:13.847394 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2fv9"] Jul 07 19:17:13.849976 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:17:13.849946 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d78dd3_0bfd_477d_96b2_435eff5b83fd.slice/crio-6c576ceb38e8fb9735f1eb298d63717125787d18209f0eefe627ee8f5fa24d6a WatchSource:0}: Error finding container 6c576ceb38e8fb9735f1eb298d63717125787d18209f0eefe627ee8f5fa24d6a: Status 404 returned error can't find the container with id 6c576ceb38e8fb9735f1eb298d63717125787d18209f0eefe627ee8f5fa24d6a Jul 07 19:17:14.174164 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:14.174131 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" event={"ID":"80d78dd3-0bfd-477d-96b2-435eff5b83fd","Type":"ContainerStarted","Data":"6c576ceb38e8fb9735f1eb298d63717125787d18209f0eefe627ee8f5fa24d6a"} Jul 07 19:17:14.251333 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:14.251303 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:14.447473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:14.447402 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:17.190159 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:17.190116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" event={"ID":"80d78dd3-0bfd-477d-96b2-435eff5b83fd","Type":"ContainerStarted","Data":"3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720"} Jul 07 19:17:17.190513 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:17.190199 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:17.211243 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:17.211189 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" podStartSLOduration=1.406692032 podStartE2EDuration="4.211175278s" totalImagesPullingTime="2.804483246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:17.209596822 +0000 UTC m=+543.145222181" watchObservedRunningTime="2026-07-07 19:17:17.211175278 +0000 UTC m=+543.146800639" Jul 07 19:17:17.548241 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:17.548214 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:17.650237 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:17.650210 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:20.138449 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:20.138416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:20.235036 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:20.235002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:28.196183 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:28.196148 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:29.035747 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:29.035711 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2fv9"] Jul 07 19:17:29.036002 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:29.035936 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" podUID="80d78dd3-0bfd-477d-96b2-435eff5b83fd" containerName="limitador" containerID="cri-o://3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720" gracePeriod=30 Jul 07 19:17:29.589962 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:29.589938 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:29.627105 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:29.627021 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80d78dd3-0bfd-477d-96b2-435eff5b83fd-config-file\") pod \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\" (UID: \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\") " Jul 07 19:17:29.627277 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:29.627146 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kpw4\" (UniqueName: \"kubernetes.io/projected/80d78dd3-0bfd-477d-96b2-435eff5b83fd-kube-api-access-8kpw4\") pod \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\" (UID: \"80d78dd3-0bfd-477d-96b2-435eff5b83fd\") " Jul 07 19:17:29.627373 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:29.627349 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d78dd3-0bfd-477d-96b2-435eff5b83fd-config-file" (OuterVolumeSpecName: "config-file") pod "80d78dd3-0bfd-477d-96b2-435eff5b83fd" (UID: "80d78dd3-0bfd-477d-96b2-435eff5b83fd"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:17:29.629416 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:29.629391 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d78dd3-0bfd-477d-96b2-435eff5b83fd-kube-api-access-8kpw4" (OuterVolumeSpecName: "kube-api-access-8kpw4") pod "80d78dd3-0bfd-477d-96b2-435eff5b83fd" (UID: "80d78dd3-0bfd-477d-96b2-435eff5b83fd"). InnerVolumeSpecName "kube-api-access-8kpw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:17:29.728276 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:29.728243 2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80d78dd3-0bfd-477d-96b2-435eff5b83fd-config-file\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:17:29.728276 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:29.728270 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kpw4\" (UniqueName: \"kubernetes.io/projected/80d78dd3-0bfd-477d-96b2-435eff5b83fd-kube-api-access-8kpw4\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:17:30.254046 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.254011 2577 generic.go:356] "Generic (PLEG): container finished" podID="80d78dd3-0bfd-477d-96b2-435eff5b83fd" containerID="3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720" exitCode=0 Jul 07 19:17:30.254193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.254072 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" Jul 07 19:17:30.254193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.254084 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" event={"ID":"80d78dd3-0bfd-477d-96b2-435eff5b83fd","Type":"ContainerDied","Data":"3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720"} Jul 07 19:17:30.254193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.254148 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2fv9" event={"ID":"80d78dd3-0bfd-477d-96b2-435eff5b83fd","Type":"ContainerDied","Data":"6c576ceb38e8fb9735f1eb298d63717125787d18209f0eefe627ee8f5fa24d6a"} Jul 07 19:17:30.254193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.254170 2577 scope.go:117] "RemoveContainer" containerID="3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720" Jul 07 19:17:30.262736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.262719 2577 scope.go:117] "RemoveContainer" containerID="3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720" Jul 07 19:17:30.262983 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:17:30.262964 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720\": container with ID starting with 3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720 not found: ID does not exist" containerID="3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720" Jul 07 19:17:30.263051 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.262991 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720"} err="failed to get container status \"3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720\": rpc error: code = NotFound desc = could not find container \"3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720\": container with ID starting with 3737ec165220d23c3f0dab6779e90d98c64faf07d09f7e09bf4ebdca45c1c720 not found: ID does not exist" Jul 07 19:17:30.276545 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.276522 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2fv9"] Jul 07 19:17:30.284801 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.284781 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2fv9"] Jul 07 19:17:30.603997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:30.603917 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80d78dd3-0bfd-477d-96b2-435eff5b83fd" path="/var/lib/kubelet/pods/80d78dd3-0bfd-477d-96b2-435eff5b83fd/volumes" Jul 07 19:17:34.882525 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.882492 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-2bhbv"] Jul 07 19:17:34.883090 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.883074 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80d78dd3-0bfd-477d-96b2-435eff5b83fd" containerName="limitador" Jul 07 19:17:34.883140 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.883095 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80d78dd3-0bfd-477d-96b2-435eff5b83fd" containerName="limitador" Jul 07 19:17:34.883217 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.883206 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80d78dd3-0bfd-477d-96b2-435eff5b83fd" containerName="limitador" Jul 07 19:17:34.887824 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.887802 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:34.891293 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.891270 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v6rqr\"" Jul 07 19:17:34.891945 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.891916 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 07 19:17:34.893199 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.893177 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-2bhbv"] Jul 07 19:17:34.920702 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.920678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:34.977415 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.977389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3adf1e3e-26fb-465c-b13c-cf417fc4461e-data\") pod \"postgres-69fb4594cf-2bhbv\" (UID: \"3adf1e3e-26fb-465c-b13c-cf417fc4461e\") " pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:34.977575 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:34.977448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx65p\" (UniqueName: \"kubernetes.io/projected/3adf1e3e-26fb-465c-b13c-cf417fc4461e-kube-api-access-nx65p\") pod \"postgres-69fb4594cf-2bhbv\" (UID: \"3adf1e3e-26fb-465c-b13c-cf417fc4461e\") " pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:35.078822 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:35.078783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3adf1e3e-26fb-465c-b13c-cf417fc4461e-data\") pod \"postgres-69fb4594cf-2bhbv\" (UID: \"3adf1e3e-26fb-465c-b13c-cf417fc4461e\") " pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:35.079028 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:35.078847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx65p\" (UniqueName: \"kubernetes.io/projected/3adf1e3e-26fb-465c-b13c-cf417fc4461e-kube-api-access-nx65p\") pod \"postgres-69fb4594cf-2bhbv\" (UID: \"3adf1e3e-26fb-465c-b13c-cf417fc4461e\") " pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:35.079171 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:35.079151 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3adf1e3e-26fb-465c-b13c-cf417fc4461e-data\") pod \"postgres-69fb4594cf-2bhbv\" (UID: \"3adf1e3e-26fb-465c-b13c-cf417fc4461e\") " pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:35.089925 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:35.089895 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx65p\" (UniqueName: \"kubernetes.io/projected/3adf1e3e-26fb-465c-b13c-cf417fc4461e-kube-api-access-nx65p\") pod \"postgres-69fb4594cf-2bhbv\" (UID: \"3adf1e3e-26fb-465c-b13c-cf417fc4461e\") " pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:35.199849 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:35.199764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:35.335616 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:35.335592 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-2bhbv"] Jul 07 19:17:35.338095 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:17:35.338069 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adf1e3e_26fb_465c_b13c_cf417fc4461e.slice/crio-a99668119cd55efef3b5fe443e8aaac3c4a0f11cfc4be10d133f814756d71196 WatchSource:0}: Error finding container a99668119cd55efef3b5fe443e8aaac3c4a0f11cfc4be10d133f814756d71196: Status 404 returned error can't find the container with id a99668119cd55efef3b5fe443e8aaac3c4a0f11cfc4be10d133f814756d71196 Jul 07 19:17:36.287604 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:36.287533 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-2bhbv" event={"ID":"3adf1e3e-26fb-465c-b13c-cf417fc4461e","Type":"ContainerStarted","Data":"a99668119cd55efef3b5fe443e8aaac3c4a0f11cfc4be10d133f814756d71196"} Jul 07 19:17:40.310093 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:40.310061 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-2bhbv" event={"ID":"3adf1e3e-26fb-465c-b13c-cf417fc4461e","Type":"ContainerStarted","Data":"062abb844397b73a99090b882fd5b134f48c0f5fbfeb6da1123f7fbbfbfa773f"} Jul 07 19:17:40.310522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:40.310184 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:40.331589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:40.331543 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-2bhbv" podStartSLOduration=1.828820113 podStartE2EDuration="6.331526083s" totalImagesPullingTime="4.50270597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:40.329896292 +0000 UTC m=+566.265521652" watchObservedRunningTime="2026-07-07 19:17:40.331526083 +0000 UTC m=+566.267151442" Jul 07 19:17:46.229025 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:46.228991 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:46.345375 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:46.345341 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-2bhbv" Jul 07 19:17:46.687322 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:46.687292 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:47.082057 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:47.082019 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:51.579483 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:51.579450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:53.950807 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:53.950748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:53.983191 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:53.983156 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55487bd746-k6t6c"] Jul 07 19:17:54.002915 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.002889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:17:54.007608 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.007588 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gkd4k\"" Jul 07 19:17:54.008357 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.008335 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 19:17:54.008488 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.008341 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 19:17:54.015891 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.011522 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55487bd746-k6t6c"] Jul 07 19:17:54.063859 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.063805 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:54.164580 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.164546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-webhook-cert\") pod \"maas-controller-55487bd746-k6t6c\" (UID: \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\") " pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:17:54.164780 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.164580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc48g\" (UniqueName: \"kubernetes.io/projected/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-kube-api-access-kc48g\") pod \"maas-controller-55487bd746-k6t6c\" (UID: \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\") " pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:17:54.165383 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.165363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:54.169480 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.169458 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9c77bd8b4-bg4fp"] Jul 07 19:17:54.172206 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.172190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:17:54.185711 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.185690 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c77bd8b4-bg4fp"] Jul 07 19:17:54.266129 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.266094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-webhook-cert\") pod \"maas-controller-55487bd746-k6t6c\" (UID: \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\") " pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:17:54.266304 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.266149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbtzf\" (UniqueName: \"kubernetes.io/projected/55220889-3c39-43aa-b3c5-c8de46f31631-kube-api-access-nbtzf\") pod \"maas-controller-9c77bd8b4-bg4fp\" (UID: \"55220889-3c39-43aa-b3c5-c8de46f31631\") " pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:17:54.266304 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.266184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc48g\" (UniqueName: \"kubernetes.io/projected/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-kube-api-access-kc48g\") pod \"maas-controller-55487bd746-k6t6c\" (UID: \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\") " pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:17:54.266304 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.266232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55220889-3c39-43aa-b3c5-c8de46f31631-webhook-cert\") pod \"maas-controller-9c77bd8b4-bg4fp\" (UID: \"55220889-3c39-43aa-b3c5-c8de46f31631\") " pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:17:54.268789 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.268763 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-webhook-cert\") pod \"maas-controller-55487bd746-k6t6c\" (UID: \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\") " pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:17:54.277303 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.277273 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc48g\" (UniqueName: \"kubernetes.io/projected/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-kube-api-access-kc48g\") pod \"maas-controller-55487bd746-k6t6c\" (UID: \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\") " pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:17:54.321166 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.321136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:17:54.367201 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.367170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55220889-3c39-43aa-b3c5-c8de46f31631-webhook-cert\") pod \"maas-controller-9c77bd8b4-bg4fp\" (UID: \"55220889-3c39-43aa-b3c5-c8de46f31631\") " pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:17:54.367357 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.367272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbtzf\" (UniqueName: \"kubernetes.io/projected/55220889-3c39-43aa-b3c5-c8de46f31631-kube-api-access-nbtzf\") pod \"maas-controller-9c77bd8b4-bg4fp\" (UID: \"55220889-3c39-43aa-b3c5-c8de46f31631\") " pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:17:54.370718 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.370467 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55220889-3c39-43aa-b3c5-c8de46f31631-webhook-cert\") pod \"maas-controller-9c77bd8b4-bg4fp\" (UID: \"55220889-3c39-43aa-b3c5-c8de46f31631\") " pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:17:54.378695 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.378670 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbtzf\" (UniqueName: \"kubernetes.io/projected/55220889-3c39-43aa-b3c5-c8de46f31631-kube-api-access-nbtzf\") pod \"maas-controller-9c77bd8b4-bg4fp\" (UID: \"55220889-3c39-43aa-b3c5-c8de46f31631\") " pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:17:54.459556 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.459536 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55487bd746-k6t6c"] Jul 07 19:17:54.461471 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:17:54.461448 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94df2cb_1f35_4f15_8d12_45c22dc06fdc.slice/crio-93dd5f60ec3a0ae086c023d18913169ca40f8be859592559610d95e485f51eca WatchSource:0}: Error finding container 93dd5f60ec3a0ae086c023d18913169ca40f8be859592559610d95e485f51eca: Status 404 returned error can't find the container with id 93dd5f60ec3a0ae086c023d18913169ca40f8be859592559610d95e485f51eca Jul 07 19:17:54.482927 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.482903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:17:54.620559 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:54.620537 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c77bd8b4-bg4fp"] Jul 07 19:17:54.622777 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:17:54.622750 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55220889_3c39_43aa_b3c5_c8de46f31631.slice/crio-e63367d561470adb70c25336f23b8e83533040d6b26e973f725f497722c811b0 WatchSource:0}: Error finding container e63367d561470adb70c25336f23b8e83533040d6b26e973f725f497722c811b0: Status 404 returned error can't find the container with id e63367d561470adb70c25336f23b8e83533040d6b26e973f725f497722c811b0 Jul 07 19:17:55.393950 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:55.393840 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55487bd746-k6t6c" event={"ID":"b94df2cb-1f35-4f15-8d12-45c22dc06fdc","Type":"ContainerStarted","Data":"93dd5f60ec3a0ae086c023d18913169ca40f8be859592559610d95e485f51eca"} Jul 07 19:17:55.396197 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:55.396166 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" event={"ID":"55220889-3c39-43aa-b3c5-c8de46f31631","Type":"ContainerStarted","Data":"e63367d561470adb70c25336f23b8e83533040d6b26e973f725f497722c811b0"} Jul 07 19:17:57.024720 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:57.024680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:57.109164 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:57.109127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:57.408085 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:57.407995 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55487bd746-k6t6c" event={"ID":"b94df2cb-1f35-4f15-8d12-45c22dc06fdc","Type":"ContainerStarted","Data":"9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9"} Jul 07 19:17:57.408273 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:57.408108 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:17:57.409501 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:57.409475 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" event={"ID":"55220889-3c39-43aa-b3c5-c8de46f31631","Type":"ContainerStarted","Data":"cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf"} Jul 07 19:17:57.409621 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:57.409585 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:17:57.432000 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:57.431955 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55487bd746-k6t6c" podStartSLOduration=2.180945259 podStartE2EDuration="4.431941944s" totalImagesPullingTime="2.250996685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:57.430047691 +0000 UTC m=+583.365673044" watchObservedRunningTime="2026-07-07 19:17:57.431941944 +0000 UTC m=+583.367567305" Jul 07 19:17:57.458184 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:17:57.458133 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" podStartSLOduration=1.357475535 podStartE2EDuration="3.458116689s" totalImagesPullingTime="2.100641154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:57.455320086 +0000 UTC m=+583.390945445" watchObservedRunningTime="2026-07-07 19:17:57.458116689 +0000 UTC m=+583.393742094" Jul 07 19:18:08.420275 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.420247 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:18:08.422058 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.422037 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:18:08.505147 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.505118 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55487bd746-k6t6c"] Jul 07 19:18:08.505325 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.505291 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-55487bd746-k6t6c" podUID="b94df2cb-1f35-4f15-8d12-45c22dc06fdc" containerName="manager" containerID="cri-o://9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9" gracePeriod=30 Jul 07 19:18:08.757315 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.757290 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:18:08.796845 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.796821 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-webhook-cert\") pod \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\" (UID: \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\") " Jul 07 19:18:08.797000 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.796910 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc48g\" (UniqueName: \"kubernetes.io/projected/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-kube-api-access-kc48g\") pod \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\" (UID: \"b94df2cb-1f35-4f15-8d12-45c22dc06fdc\") " Jul 07 19:18:08.799155 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.799128 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b94df2cb-1f35-4f15-8d12-45c22dc06fdc" (UID: "b94df2cb-1f35-4f15-8d12-45c22dc06fdc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:18:08.799243 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.799209 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-kube-api-access-kc48g" (OuterVolumeSpecName: "kube-api-access-kc48g") pod "b94df2cb-1f35-4f15-8d12-45c22dc06fdc" (UID: "b94df2cb-1f35-4f15-8d12-45c22dc06fdc"). InnerVolumeSpecName "kube-api-access-kc48g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:18:08.897699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.897668 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc48g\" (UniqueName: \"kubernetes.io/projected/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-kube-api-access-kc48g\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:18:08.897699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:08.897694 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b94df2cb-1f35-4f15-8d12-45c22dc06fdc-webhook-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:18:09.469129 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:09.469090 2577 generic.go:356] "Generic (PLEG): container finished" podID="b94df2cb-1f35-4f15-8d12-45c22dc06fdc" containerID="9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9" exitCode=0 Jul 07 19:18:09.469612 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:09.469154 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55487bd746-k6t6c" Jul 07 19:18:09.469612 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:09.469182 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55487bd746-k6t6c" event={"ID":"b94df2cb-1f35-4f15-8d12-45c22dc06fdc","Type":"ContainerDied","Data":"9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9"} Jul 07 19:18:09.469612 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:09.469284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55487bd746-k6t6c" event={"ID":"b94df2cb-1f35-4f15-8d12-45c22dc06fdc","Type":"ContainerDied","Data":"93dd5f60ec3a0ae086c023d18913169ca40f8be859592559610d95e485f51eca"} Jul 07 19:18:09.469612 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:09.469317 2577 scope.go:117] "RemoveContainer" containerID="9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9" Jul 07 19:18:09.478655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:09.478638 2577 scope.go:117] "RemoveContainer" containerID="9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9" Jul 07 19:18:09.478909 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:18:09.478892 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9\": container with ID starting with 9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9 not found: ID does not exist" containerID="9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9" Jul 07 19:18:09.478978 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:09.478916 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9"} err="failed to get container status \"9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9\": rpc error: code = NotFound desc = could not find container \"9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9\": container with ID starting with 9fe51c851a6257dbf5effeaf24404cdf3ef8aa2c041ac9bbec3069091b9d47e9 not found: ID does not exist" Jul 07 19:18:09.492428 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:09.492406 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55487bd746-k6t6c"] Jul 07 19:18:09.497764 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:09.497745 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55487bd746-k6t6c"] Jul 07 19:18:10.603841 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:10.603808 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94df2cb-1f35-4f15-8d12-45c22dc06fdc" path="/var/lib/kubelet/pods/b94df2cb-1f35-4f15-8d12-45c22dc06fdc/volumes" Jul 07 19:18:29.543186 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:29.543151 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:30.063593 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.063556 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-6d8c95d7dc-4wkl6"] Jul 07 19:18:30.063997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.063985 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94df2cb-1f35-4f15-8d12-45c22dc06fdc" containerName="manager" Jul 07 19:18:30.064056 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.063999 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94df2cb-1f35-4f15-8d12-45c22dc06fdc" containerName="manager" Jul 07 19:18:30.064091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.064055 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94df2cb-1f35-4f15-8d12-45c22dc06fdc" containerName="manager" Jul 07 19:18:30.067120 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.067099 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.070569 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.070546 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 19:18:30.070683 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.070596 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tdpwr\"" Jul 07 19:18:30.086188 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.086162 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6d8c95d7dc-4wkl6"] Jul 07 19:18:30.209473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.209439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cczs5\" (UniqueName: \"kubernetes.io/projected/f3e3e2ee-1ea8-4687-a389-bcc6e10515b4-kube-api-access-cczs5\") pod \"maas-api-6d8c95d7dc-4wkl6\" (UID: \"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4\") " pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.209660 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.209505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3e3e2ee-1ea8-4687-a389-bcc6e10515b4-maas-api-tls\") pod \"maas-api-6d8c95d7dc-4wkl6\" (UID: \"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4\") " pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.209660 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.209548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e3e2ee-1ea8-4687-a389-bcc6e10515b4-openshift-service-ca\") pod \"maas-api-6d8c95d7dc-4wkl6\" (UID: \"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4\") " pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.310767 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.310729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cczs5\" (UniqueName: \"kubernetes.io/projected/f3e3e2ee-1ea8-4687-a389-bcc6e10515b4-kube-api-access-cczs5\") pod \"maas-api-6d8c95d7dc-4wkl6\" (UID: \"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4\") " pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.310990 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.310788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3e3e2ee-1ea8-4687-a389-bcc6e10515b4-maas-api-tls\") pod \"maas-api-6d8c95d7dc-4wkl6\" (UID: \"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4\") " pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.310990 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.310822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e3e2ee-1ea8-4687-a389-bcc6e10515b4-openshift-service-ca\") pod \"maas-api-6d8c95d7dc-4wkl6\" (UID: \"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4\") " pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.311520 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.311495 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e3e2ee-1ea8-4687-a389-bcc6e10515b4-openshift-service-ca\") pod \"maas-api-6d8c95d7dc-4wkl6\" (UID: \"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4\") " pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.313371 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.313345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3e3e2ee-1ea8-4687-a389-bcc6e10515b4-maas-api-tls\") pod \"maas-api-6d8c95d7dc-4wkl6\" (UID: \"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4\") " pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.327401 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.327337 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cczs5\" (UniqueName: \"kubernetes.io/projected/f3e3e2ee-1ea8-4687-a389-bcc6e10515b4-kube-api-access-cczs5\") pod \"maas-api-6d8c95d7dc-4wkl6\" (UID: \"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4\") " pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.377570 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.377536 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:30.525023 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.524914 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6d8c95d7dc-4wkl6"] Jul 07 19:18:30.527977 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:18:30.527953 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e3e2ee_1ea8_4687_a389_bcc6e10515b4.slice/crio-476e141e94c851e0cd57fd8e2c019e7c026c2bd9509a1c056b7771efa8b488e4 WatchSource:0}: Error finding container 476e141e94c851e0cd57fd8e2c019e7c026c2bd9509a1c056b7771efa8b488e4: Status 404 returned error can't find the container with id 476e141e94c851e0cd57fd8e2c019e7c026c2bd9509a1c056b7771efa8b488e4 Jul 07 19:18:30.576596 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.576551 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" event={"ID":"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4","Type":"ContainerStarted","Data":"476e141e94c851e0cd57fd8e2c019e7c026c2bd9509a1c056b7771efa8b488e4"} Jul 07 19:18:30.805855 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.805821 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-chkp7"] Jul 07 19:18:30.812811 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.812788 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:30.818697 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.818670 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fqw78\"" Jul 07 19:18:30.818861 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.818673 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 07 19:18:30.823760 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.823735 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-chkp7"] Jul 07 19:18:30.917312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.917274 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qmhs\" (UniqueName: \"kubernetes.io/projected/3c2fb729-32ba-4b76-acb7-ab6c74fe1074-kube-api-access-6qmhs\") pod \"payload-pre-processing-7964457696-chkp7\" (UID: \"3c2fb729-32ba-4b76-acb7-ab6c74fe1074\") " pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:30.917505 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.917442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2fb729-32ba-4b76-acb7-ab6c74fe1074-plugins-config-volume\") pod \"payload-pre-processing-7964457696-chkp7\" (UID: \"3c2fb729-32ba-4b76-acb7-ab6c74fe1074\") " pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:30.935576 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:30.935541 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:31.018801 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:31.018765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qmhs\" (UniqueName: \"kubernetes.io/projected/3c2fb729-32ba-4b76-acb7-ab6c74fe1074-kube-api-access-6qmhs\") pod \"payload-pre-processing-7964457696-chkp7\" (UID: \"3c2fb729-32ba-4b76-acb7-ab6c74fe1074\") " pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:31.019003 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:31.018850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2fb729-32ba-4b76-acb7-ab6c74fe1074-plugins-config-volume\") pod \"payload-pre-processing-7964457696-chkp7\" (UID: \"3c2fb729-32ba-4b76-acb7-ab6c74fe1074\") " pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:31.019575 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:31.019556 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2fb729-32ba-4b76-acb7-ab6c74fe1074-plugins-config-volume\") pod \"payload-pre-processing-7964457696-chkp7\" (UID: \"3c2fb729-32ba-4b76-acb7-ab6c74fe1074\") " pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:31.028513 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:31.028485 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qmhs\" (UniqueName: \"kubernetes.io/projected/3c2fb729-32ba-4b76-acb7-ab6c74fe1074-kube-api-access-6qmhs\") pod \"payload-pre-processing-7964457696-chkp7\" (UID: \"3c2fb729-32ba-4b76-acb7-ab6c74fe1074\") " pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:31.125232 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:31.125139 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:31.275584 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:31.275555 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-chkp7"] Jul 07 19:18:31.279105 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:18:31.279077 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2fb729_32ba_4b76_acb7_ab6c74fe1074.slice/crio-77be2b8c24968c7669d28cea963c520273183ce22b57a4cbe330ffeeed0a6844 WatchSource:0}: Error finding container 77be2b8c24968c7669d28cea963c520273183ce22b57a4cbe330ffeeed0a6844: Status 404 returned error can't find the container with id 77be2b8c24968c7669d28cea963c520273183ce22b57a4cbe330ffeeed0a6844 Jul 07 19:18:31.593548 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:31.593467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" event={"ID":"3c2fb729-32ba-4b76-acb7-ab6c74fe1074","Type":"ContainerStarted","Data":"77be2b8c24968c7669d28cea963c520273183ce22b57a4cbe330ffeeed0a6844"} Jul 07 19:18:33.627932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:33.627898 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" event={"ID":"3c2fb729-32ba-4b76-acb7-ab6c74fe1074","Type":"ContainerStarted","Data":"6f5cf4d1baa4cdf3525c1ef72726154fbf11f5d1cef881559468c41c5a2d560b"} Jul 07 19:18:33.628431 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:33.628090 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:33.629434 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:33.629410 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" event={"ID":"f3e3e2ee-1ea8-4687-a389-bcc6e10515b4","Type":"ContainerStarted","Data":"4ce67f89885757a2410f3d875d46d1bec7200a9eb7efa21e0ab5622da20c3273"} Jul 07 19:18:33.629571 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:33.629551 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:33.652113 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:33.652069 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" podStartSLOduration=1.863846304 podStartE2EDuration="3.652055281s" totalImagesPullingTime="1.788208977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:18:33.647707805 +0000 UTC m=+619.583333164" watchObservedRunningTime="2026-07-07 19:18:33.652055281 +0000 UTC m=+619.587680639" Jul 07 19:18:33.675989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:33.675943 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" podStartSLOduration=1.137375071 podStartE2EDuration="3.675928378s" totalImagesPullingTime="2.538553307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:18:33.674363202 +0000 UTC m=+619.609988567" watchObservedRunningTime="2026-07-07 19:18:33.675928378 +0000 UTC m=+619.611553737" Jul 07 19:18:38.660183 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:38.660151 2577 generic.go:356] "Generic (PLEG): container finished" podID="93df3e10-d156-40e0-8f81-367099052b5f" containerID="059064c939f107614ff98d0d34d2e40081a4dfffeb750b314acd85f2eff69489" exitCode=0 Jul 07 19:18:38.660723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:38.660233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" event={"ID":"93df3e10-d156-40e0-8f81-367099052b5f","Type":"ContainerDied","Data":"059064c939f107614ff98d0d34d2e40081a4dfffeb750b314acd85f2eff69489"} Jul 07 19:18:38.660790 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:38.660754 2577 scope.go:117] "RemoveContainer" containerID="059064c939f107614ff98d0d34d2e40081a4dfffeb750b314acd85f2eff69489" Jul 07 19:18:38.661807 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:38.661778 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:18:38.661868 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:38.661846 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:18:38.661933 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:38.661899 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:18:39.639519 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:39.639492 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-6d8c95d7dc-4wkl6" Jul 07 19:18:39.667059 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:39.667030 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" event={"ID":"93df3e10-d156-40e0-8f81-367099052b5f","Type":"ContainerStarted","Data":"e658660c31fd3c1cd95d039c7e67dbc27500b31748b4f144764d17740d74fd14"} Jul 07 19:18:39.757693 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:39.757661 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:18:39.758770 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:39.758742 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-d6nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 07 19:18:39.758913 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:39.758789 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" podUID="93df3e10-d156-40e0-8f81-367099052b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 07 19:18:40.715614 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:40.715579 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:40.758579 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:40.758542 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-d6nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 07 19:18:40.758769 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:40.758600 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" podUID="93df3e10-d156-40e0-8f81-367099052b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 07 19:18:40.815056 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:40.815019 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:41.757815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:41.757776 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-d6nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 07 19:18:41.758239 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:41.757833 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" podUID="93df3e10-d156-40e0-8f81-367099052b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 07 19:18:42.758360 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:42.758325 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-d6nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 07 19:18:42.758759 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:42.758378 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" podUID="93df3e10-d156-40e0-8f81-367099052b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 07 19:18:43.168434 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:43.168351 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:18:43.762099 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:43.762073 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:18:43.763203 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:43.763183 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-d6nj7" Jul 07 19:18:44.636040 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:44.636014 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-chkp7" Jul 07 19:18:46.929918 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.929856 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd"] Jul 07 19:18:46.932900 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.932860 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:46.935979 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.935957 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 19:18:46.936787 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.936760 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n7c64\"" Jul 07 19:18:46.937039 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.936785 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 19:18:46.937039 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.936767 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 19:18:46.945555 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.945532 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd"] Jul 07 19:18:46.974329 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.974302 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:46.983432 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.983400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:46.983594 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.983491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00fc2de-eb4b-45de-a766-14cc32b41de6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:46.983594 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.983534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:46.983594 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.983566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:46.983737 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.983600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5chx\" (UniqueName: \"kubernetes.io/projected/c00fc2de-eb4b-45de-a766-14cc32b41de6-kube-api-access-z5chx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:46.983737 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:46.983677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.084248 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.084206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.084248 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.084250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.084493 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.084274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5chx\" (UniqueName: \"kubernetes.io/projected/c00fc2de-eb4b-45de-a766-14cc32b41de6-kube-api-access-z5chx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.084493 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.084312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.084493 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.084378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.084493 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.084458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00fc2de-eb4b-45de-a766-14cc32b41de6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.084777 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.084750 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.084854 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.084790 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.084956 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.084936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.086690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.086667 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00fc2de-eb4b-45de-a766-14cc32b41de6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.087125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.087104 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00fc2de-eb4b-45de-a766-14cc32b41de6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.094086 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.094060 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5chx\" (UniqueName: \"kubernetes.io/projected/c00fc2de-eb4b-45de-a766-14cc32b41de6-kube-api-access-z5chx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd\" (UID: \"c00fc2de-eb4b-45de-a766-14cc32b41de6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.245426 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.245387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:47.391864 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.391842 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd"] Jul 07 19:18:47.394158 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:18:47.394115 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00fc2de_eb4b_45de_a766_14cc32b41de6.slice/crio-d851a7a654d2ae68ec60ccba2b880e8cb02508db2003725690cbb5deb6ffb977 WatchSource:0}: Error finding container d851a7a654d2ae68ec60ccba2b880e8cb02508db2003725690cbb5deb6ffb977: Status 404 returned error can't find the container with id d851a7a654d2ae68ec60ccba2b880e8cb02508db2003725690cbb5deb6ffb977 Jul 07 19:18:47.712016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:47.711901 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" event={"ID":"c00fc2de-eb4b-45de-a766-14cc32b41de6","Type":"ContainerStarted","Data":"d851a7a654d2ae68ec60ccba2b880e8cb02508db2003725690cbb5deb6ffb977"} Jul 07 19:18:48.718627 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:48.718592 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" event={"ID":"c00fc2de-eb4b-45de-a766-14cc32b41de6","Type":"ContainerStarted","Data":"372e28c4add097a54c83f597879bebf5b78394b152bba4765b7ed52f255286f0"} Jul 07 19:18:48.740488 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:48.740429 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" podStartSLOduration=1.6028102039999998 podStartE2EDuration="2.740409176s" totalImagesPullingTime="1.137598972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:18:48.740385458 +0000 UTC m=+634.676010819" watchObservedRunningTime="2026-07-07 19:18:48.740409176 +0000 UTC m=+634.676034535" Jul 07 19:18:51.169385 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.169353 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd"] Jul 07 19:18:51.172975 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.172956 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.177755 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.177732 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 07 19:18:51.192248 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.192221 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd"] Jul 07 19:18:51.205643 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.205605 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:51.224825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.224791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2w4b\" (UniqueName: \"kubernetes.io/projected/c968b2e1-f195-4583-b42e-3771722f8b5c-kube-api-access-r2w4b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.225015 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.224903 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.225015 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.224922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.225015 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.224955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.225192 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.225077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.225192 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.225116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c968b2e1-f195-4583-b42e-3771722f8b5c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.326546 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.326512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.326723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.326574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.326723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.326606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c968b2e1-f195-4583-b42e-3771722f8b5c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.326723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.326639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2w4b\" (UniqueName: \"kubernetes.io/projected/c968b2e1-f195-4583-b42e-3771722f8b5c-kube-api-access-r2w4b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.326970 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.326732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.326970 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.326756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.327077 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.326978 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.327077 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.327026 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.327379 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.327320 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.329148 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.329114 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c968b2e1-f195-4583-b42e-3771722f8b5c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.329375 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.329354 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c968b2e1-f195-4583-b42e-3771722f8b5c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.336684 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.336665 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2w4b\" (UniqueName: \"kubernetes.io/projected/c968b2e1-f195-4583-b42e-3771722f8b5c-kube-api-access-r2w4b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fbcxd\" (UID: \"c968b2e1-f195-4583-b42e-3771722f8b5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.483821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.483736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:18:51.636728 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.636668 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd"] Jul 07 19:18:51.639077 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:18:51.639048 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc968b2e1_f195_4583_b42e_3771722f8b5c.slice/crio-9928b5e4839dfb2b682d1772b1c5699ca6db0ef1bb1a5ba6392972805b0ad026 WatchSource:0}: Error finding container 9928b5e4839dfb2b682d1772b1c5699ca6db0ef1bb1a5ba6392972805b0ad026: Status 404 returned error can't find the container with id 9928b5e4839dfb2b682d1772b1c5699ca6db0ef1bb1a5ba6392972805b0ad026 Jul 07 19:18:51.640457 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.640438 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:18:51.735233 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:51.735141 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" event={"ID":"c968b2e1-f195-4583-b42e-3771722f8b5c","Type":"ContainerStarted","Data":"9928b5e4839dfb2b682d1772b1c5699ca6db0ef1bb1a5ba6392972805b0ad026"} Jul 07 19:18:52.741423 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:52.741386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" event={"ID":"c968b2e1-f195-4583-b42e-3771722f8b5c","Type":"ContainerStarted","Data":"56dbb1bc0d728bf9a21ba0b2687b2e6ff92344b9f8e2e51262127abc70335a01"} Jul 07 19:18:52.764514 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:52.764459 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" podStartSLOduration=1.563922313 podStartE2EDuration="1.764438552s" totalImagesPullingTime="200.516239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:18:52.763476808 +0000 UTC m=+638.699102167" watchObservedRunningTime="2026-07-07 19:18:52.764438552 +0000 UTC m=+638.700063915" Jul 07 19:18:57.246312 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:57.246276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:57.246704 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:57.246342 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:57.258753 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:57.258721 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:57.776576 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:57.776547 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd" Jul 07 19:18:58.812561 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:18:58.812514 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:01.484830 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:19:01.484796 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:19:01.485331 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:19:01.484856 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:19:01.497630 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:19:01.497598 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:19:01.797496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:19:01.797467 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fbcxd" Jul 07 19:19:02.650446 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:19:02.650394 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:09.811404 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:19:09.811374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:20.099984 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:19:20.099944 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:31.920333 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:19:31.920280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:50.325181 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:19:50.325140 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:20:28.388078 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:20:28.387996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:20:28.516311 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:20:28.516281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:20:29.547368 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:20:29.547335 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:20:29.776516 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:20:29.776483 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:21:40.431679 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.431596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:21:40.519458 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.519424 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9c77bd8b4-bg4fp"] Jul 07 19:21:40.519692 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.519665 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" podUID="55220889-3c39-43aa-b3c5-c8de46f31631" containerName="manager" containerID="cri-o://cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf" gracePeriod=30 Jul 07 19:21:40.632068 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.632034 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:21:40.773779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.773758 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:21:40.865333 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.865306 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbtzf\" (UniqueName: \"kubernetes.io/projected/55220889-3c39-43aa-b3c5-c8de46f31631-kube-api-access-nbtzf\") pod \"55220889-3c39-43aa-b3c5-c8de46f31631\" (UID: \"55220889-3c39-43aa-b3c5-c8de46f31631\") " Jul 07 19:21:40.865496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.865404 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55220889-3c39-43aa-b3c5-c8de46f31631-webhook-cert\") pod \"55220889-3c39-43aa-b3c5-c8de46f31631\" (UID: \"55220889-3c39-43aa-b3c5-c8de46f31631\") " Jul 07 19:21:40.867455 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.867422 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55220889-3c39-43aa-b3c5-c8de46f31631-kube-api-access-nbtzf" (OuterVolumeSpecName: "kube-api-access-nbtzf") pod "55220889-3c39-43aa-b3c5-c8de46f31631" (UID: "55220889-3c39-43aa-b3c5-c8de46f31631"). InnerVolumeSpecName "kube-api-access-nbtzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:21:40.867455 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.867436 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55220889-3c39-43aa-b3c5-c8de46f31631-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "55220889-3c39-43aa-b3c5-c8de46f31631" (UID: "55220889-3c39-43aa-b3c5-c8de46f31631"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:21:40.966481 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.966400 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55220889-3c39-43aa-b3c5-c8de46f31631-webhook-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:21:40.966481 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:40.966429 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbtzf\" (UniqueName: \"kubernetes.io/projected/55220889-3c39-43aa-b3c5-c8de46f31631-kube-api-access-nbtzf\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:21:41.600185 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:41.600154 2577 generic.go:356] "Generic (PLEG): container finished" podID="55220889-3c39-43aa-b3c5-c8de46f31631" containerID="cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf" exitCode=0 Jul 07 19:21:41.600696 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:41.600198 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" Jul 07 19:21:41.600696 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:41.600251 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" event={"ID":"55220889-3c39-43aa-b3c5-c8de46f31631","Type":"ContainerDied","Data":"cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf"} Jul 07 19:21:41.600696 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:41.600310 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c77bd8b4-bg4fp" event={"ID":"55220889-3c39-43aa-b3c5-c8de46f31631","Type":"ContainerDied","Data":"e63367d561470adb70c25336f23b8e83533040d6b26e973f725f497722c811b0"} Jul 07 19:21:41.600696 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:41.600346 2577 scope.go:117] "RemoveContainer" containerID="cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf" Jul 07 19:21:41.610665 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:41.610649 2577 scope.go:117] "RemoveContainer" containerID="cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf" Jul 07 19:21:41.610952 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:21:41.610933 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf\": container with ID starting with cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf not found: ID does not exist" containerID="cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf" Jul 07 19:21:41.611015 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:41.610962 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf"} err="failed to get container status \"cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf\": rpc error: code = NotFound desc = could not find container \"cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf\": container with ID starting with cb7de10065544eb76dcae717f23bc0a94c155234434274b9891fb9ce4955c0bf not found: ID does not exist" Jul 07 19:21:41.632067 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:41.632041 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9c77bd8b4-bg4fp"] Jul 07 19:21:41.635989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:41.635954 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9c77bd8b4-bg4fp"] Jul 07 19:21:42.364152 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.364117 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:21:42.465068 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.465034 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9c77bd8b4-lpqf7"] Jul 07 19:21:42.465461 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.465441 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55220889-3c39-43aa-b3c5-c8de46f31631" containerName="manager" Jul 07 19:21:42.465510 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.465465 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="55220889-3c39-43aa-b3c5-c8de46f31631" containerName="manager" Jul 07 19:21:42.465550 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.465544 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="55220889-3c39-43aa-b3c5-c8de46f31631" containerName="manager" Jul 07 19:21:42.469849 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.469831 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:21:42.474093 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.474066 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gkd4k\"" Jul 07 19:21:42.474215 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.474131 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 19:21:42.474215 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.474082 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 19:21:42.485335 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.485312 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c77bd8b4-lpqf7"] Jul 07 19:21:42.582668 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.582635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m76n\" (UniqueName: \"kubernetes.io/projected/acbc8b6a-958b-483e-b393-1d6b17ed660f-kube-api-access-9m76n\") pod \"maas-controller-9c77bd8b4-lpqf7\" (UID: \"acbc8b6a-958b-483e-b393-1d6b17ed660f\") " pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:21:42.582836 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.582699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acbc8b6a-958b-483e-b393-1d6b17ed660f-webhook-cert\") pod \"maas-controller-9c77bd8b4-lpqf7\" (UID: \"acbc8b6a-958b-483e-b393-1d6b17ed660f\") " pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:21:42.605210 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.605183 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55220889-3c39-43aa-b3c5-c8de46f31631" path="/var/lib/kubelet/pods/55220889-3c39-43aa-b3c5-c8de46f31631/volumes" Jul 07 19:21:42.683345 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.683256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acbc8b6a-958b-483e-b393-1d6b17ed660f-webhook-cert\") pod \"maas-controller-9c77bd8b4-lpqf7\" (UID: \"acbc8b6a-958b-483e-b393-1d6b17ed660f\") " pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:21:42.683500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.683352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m76n\" (UniqueName: \"kubernetes.io/projected/acbc8b6a-958b-483e-b393-1d6b17ed660f-kube-api-access-9m76n\") pod \"maas-controller-9c77bd8b4-lpqf7\" (UID: \"acbc8b6a-958b-483e-b393-1d6b17ed660f\") " pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:21:42.686109 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.686086 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acbc8b6a-958b-483e-b393-1d6b17ed660f-webhook-cert\") pod \"maas-controller-9c77bd8b4-lpqf7\" (UID: \"acbc8b6a-958b-483e-b393-1d6b17ed660f\") " pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:21:42.693401 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.693376 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m76n\" (UniqueName: \"kubernetes.io/projected/acbc8b6a-958b-483e-b393-1d6b17ed660f-kube-api-access-9m76n\") pod \"maas-controller-9c77bd8b4-lpqf7\" (UID: \"acbc8b6a-958b-483e-b393-1d6b17ed660f\") " pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:21:42.779866 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.779830 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:21:42.920738 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:42.920714 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9c77bd8b4-lpqf7"] Jul 07 19:21:42.922889 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:21:42.922847 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbc8b6a_958b_483e_b393_1d6b17ed660f.slice/crio-143f579d24151088697a7bb4aee0c1c4302079f5e15f941eb5a8aa028b8c20db WatchSource:0}: Error finding container 143f579d24151088697a7bb4aee0c1c4302079f5e15f941eb5a8aa028b8c20db: Status 404 returned error can't find the container with id 143f579d24151088697a7bb4aee0c1c4302079f5e15f941eb5a8aa028b8c20db Jul 07 19:21:43.620982 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:43.620866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" event={"ID":"acbc8b6a-958b-483e-b393-1d6b17ed660f","Type":"ContainerStarted","Data":"aa45873f0d463422bd817fda9dcba159e1932d42f7f8d0beba772a15da429b0c"} Jul 07 19:21:43.621366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:43.620990 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:21:43.621366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:43.621009 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" event={"ID":"acbc8b6a-958b-483e-b393-1d6b17ed660f","Type":"ContainerStarted","Data":"143f579d24151088697a7bb4aee0c1c4302079f5e15f941eb5a8aa028b8c20db"} Jul 07 19:21:43.644840 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:43.644798 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" podStartSLOduration=1.224823241 podStartE2EDuration="1.644785099s" totalImagesPullingTime="419.961858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:21:43.642493162 +0000 UTC m=+809.578118533" watchObservedRunningTime="2026-07-07 19:21:43.644785099 +0000 UTC m=+809.580410501" Jul 07 19:21:54.633000 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:21:54.632968 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9c77bd8b4-lpqf7" Jul 07 19:22:25.171925 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:22:25.171841 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:25.274992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:22:25.274959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:25.405234 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:22:25.405173 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:32.151930 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:22:32.151889 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:32.268369 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:22:32.268337 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:32.390901 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:22:32.390859 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:32.534981 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:22:32.534944 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:38.930465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:22:38.930435 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:39.035585 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:22:39.035551 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:23:18.813271 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:23:18.813235 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:23:18.913227 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:23:18.913197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:56.651831 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:24:56.651730 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:57.714539 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:24:57.714478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:25:07.108399 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:25:07.108358 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:37.901179 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:37.901145 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc"] Jul 07 19:26:37.901782 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:37.901437 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" podUID="28159cac-8ea7-4248-8de5-90290ec0e51e" containerName="manager" containerID="cri-o://1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a" gracePeriod=10 Jul 07 19:26:41.554962 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:41.554932 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:26:41.684384 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:41.684295 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28159cac-8ea7-4248-8de5-90290ec0e51e-extensions-socket-volume\") pod \"28159cac-8ea7-4248-8de5-90290ec0e51e\" (UID: \"28159cac-8ea7-4248-8de5-90290ec0e51e\") " Jul 07 19:26:41.684530 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:41.684405 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4vdm\" (UniqueName: \"kubernetes.io/projected/28159cac-8ea7-4248-8de5-90290ec0e51e-kube-api-access-n4vdm\") pod \"28159cac-8ea7-4248-8de5-90290ec0e51e\" (UID: \"28159cac-8ea7-4248-8de5-90290ec0e51e\") " Jul 07 19:26:41.684736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:41.684711 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28159cac-8ea7-4248-8de5-90290ec0e51e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "28159cac-8ea7-4248-8de5-90290ec0e51e" (UID: "28159cac-8ea7-4248-8de5-90290ec0e51e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:41.686506 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:41.686483 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28159cac-8ea7-4248-8de5-90290ec0e51e-kube-api-access-n4vdm" (OuterVolumeSpecName: "kube-api-access-n4vdm") pod "28159cac-8ea7-4248-8de5-90290ec0e51e" (UID: "28159cac-8ea7-4248-8de5-90290ec0e51e"). InnerVolumeSpecName "kube-api-access-n4vdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:26:41.785335 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:41.785306 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4vdm\" (UniqueName: \"kubernetes.io/projected/28159cac-8ea7-4248-8de5-90290ec0e51e-kube-api-access-n4vdm\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:26:41.785335 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:41.785330 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28159cac-8ea7-4248-8de5-90290ec0e51e-extensions-socket-volume\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:26:42.157703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.157675 2577 generic.go:356] "Generic (PLEG): container finished" podID="28159cac-8ea7-4248-8de5-90290ec0e51e" containerID="1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a" exitCode=0 Jul 07 19:26:42.157901 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.157720 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" Jul 07 19:26:42.157901 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.157740 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" event={"ID":"28159cac-8ea7-4248-8de5-90290ec0e51e","Type":"ContainerDied","Data":"1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a"} Jul 07 19:26:42.157901 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.157780 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc" event={"ID":"28159cac-8ea7-4248-8de5-90290ec0e51e","Type":"ContainerDied","Data":"01243a89d24f336249349c3d69d2592406f5de4331cc86c7d2fb4ce91db74b7a"} Jul 07 19:26:42.157901 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.157800 2577 scope.go:117] "RemoveContainer" containerID="1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a" Jul 07 19:26:42.167891 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.167859 2577 scope.go:117] "RemoveContainer" containerID="1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a" Jul 07 19:26:42.168113 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:26:42.168095 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a\": container with ID starting with 1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a not found: ID does not exist" containerID="1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a" Jul 07 19:26:42.168167 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.168119 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a"} err="failed to get container status \"1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a\": rpc error: code = NotFound desc = could not find container \"1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a\": container with ID starting with 1692a1a6a900fb723add9a4c7cc29abd3db0bb7c2059869ef0e82492b0d4253a not found: ID does not exist" Jul 07 19:26:42.183214 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.183194 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc"] Jul 07 19:26:42.194251 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.194231 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqqmc"] Jul 07 19:26:42.604557 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:26:42.604528 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28159cac-8ea7-4248-8de5-90290ec0e51e" path="/var/lib/kubelet/pods/28159cac-8ea7-4248-8de5-90290ec0e51e/volumes" Jul 07 19:27:31.695506 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:31.695471 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:39.171289 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.171213 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5"] Jul 07 19:27:39.171714 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.171644 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28159cac-8ea7-4248-8de5-90290ec0e51e" containerName="manager" Jul 07 19:27:39.171714 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.171656 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="28159cac-8ea7-4248-8de5-90290ec0e51e" containerName="manager" Jul 07 19:27:39.171786 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.171723 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="28159cac-8ea7-4248-8de5-90290ec0e51e" containerName="manager" Jul 07 19:27:39.176084 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.176068 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:27:39.178964 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.178938 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hgr87\"" Jul 07 19:27:39.188934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.188912 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5"] Jul 07 19:27:39.318487 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.318457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0341e53b-1ea6-4ffa-a235-64bc0c097091-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vmgx5\" (UID: \"0341e53b-1ea6-4ffa-a235-64bc0c097091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:27:39.318668 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.318514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prw5h\" (UniqueName: \"kubernetes.io/projected/0341e53b-1ea6-4ffa-a235-64bc0c097091-kube-api-access-prw5h\") pod \"kuadrant-operator-controller-manager-86878c769f-vmgx5\" (UID: \"0341e53b-1ea6-4ffa-a235-64bc0c097091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:27:39.419168 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.419120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0341e53b-1ea6-4ffa-a235-64bc0c097091-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vmgx5\" (UID: \"0341e53b-1ea6-4ffa-a235-64bc0c097091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:27:39.419340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.419198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prw5h\" (UniqueName: \"kubernetes.io/projected/0341e53b-1ea6-4ffa-a235-64bc0c097091-kube-api-access-prw5h\") pod \"kuadrant-operator-controller-manager-86878c769f-vmgx5\" (UID: \"0341e53b-1ea6-4ffa-a235-64bc0c097091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:27:39.419511 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.419491 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0341e53b-1ea6-4ffa-a235-64bc0c097091-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vmgx5\" (UID: \"0341e53b-1ea6-4ffa-a235-64bc0c097091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:27:39.429255 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.429190 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prw5h\" (UniqueName: \"kubernetes.io/projected/0341e53b-1ea6-4ffa-a235-64bc0c097091-kube-api-access-prw5h\") pod \"kuadrant-operator-controller-manager-86878c769f-vmgx5\" (UID: \"0341e53b-1ea6-4ffa-a235-64bc0c097091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:27:39.486989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.486958 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:27:39.627123 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:39.627100 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5"] Jul 07 19:27:39.628956 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:27:39.628935 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0341e53b_1ea6_4ffa_a235_64bc0c097091.slice/crio-5c5edcf6b0954afe21db85830e3834e562cbffc21fefd35f1edd8047a04c8bc6 WatchSource:0}: Error finding container 5c5edcf6b0954afe21db85830e3834e562cbffc21fefd35f1edd8047a04c8bc6: Status 404 returned error can't find the container with id 5c5edcf6b0954afe21db85830e3834e562cbffc21fefd35f1edd8047a04c8bc6 Jul 07 19:27:40.456814 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:40.456782 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" event={"ID":"0341e53b-1ea6-4ffa-a235-64bc0c097091","Type":"ContainerStarted","Data":"568d7da17a730d1459e4c01731443a6712994fb99c2c80234bfa3a7b5227ec07"} Jul 07 19:27:40.457213 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:40.456831 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" event={"ID":"0341e53b-1ea6-4ffa-a235-64bc0c097091","Type":"ContainerStarted","Data":"5c5edcf6b0954afe21db85830e3834e562cbffc21fefd35f1edd8047a04c8bc6"} Jul 07 19:27:40.457213 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:40.456895 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:27:40.485141 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:40.485084 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" podStartSLOduration=1.485069106 podStartE2EDuration="1.485069106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:27:40.480989811 +0000 UTC m=+1166.416615169" watchObservedRunningTime="2026-07-07 19:27:40.485069106 +0000 UTC m=+1166.420694466" Jul 07 19:27:51.464321 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:27:51.464287 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmgx5" Jul 07 19:30:15.659435 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:30:15.659403 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:30:16.926327 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:30:16.926278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:28.620206 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:33:28.620168 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:30.132384 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:33:30.132348 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:39.537267 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:33:39.537175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:47.789549 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:33:47.789512 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:52.896181 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:33:52.896140 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:55.811624 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:33:55.811561 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:56.038795 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:33:56.038760 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:56.302065 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:33:56.302039 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:02.512033 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.511994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:02.614218 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.614180 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:02.688848 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.688817 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh"] Jul 07 19:34:02.693291 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.693272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.696154 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.696135 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8b7b410f-openshift-default-dockercfg-fkszx\"" Jul 07 19:34:02.714373 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.714338 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh"] Jul 07 19:34:02.722007 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.721981 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:02.762930 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.762857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-data\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.762930 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.762922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-token\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.763092 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.762957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-podinfo\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.763092 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.762985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istiod-ca-cert\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.763092 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.763059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-envoy\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.763285 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.763095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-credential-socket\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.763285 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.763237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-socket\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.763398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.763283 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzj6k\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-kube-api-access-hzj6k\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.763398 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.763325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-certs\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.822664 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.822631 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:02.864350 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-credential-socket\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-socket\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzj6k\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-kube-api-access-hzj6k\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-certs\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-data\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-token\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-podinfo\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istiod-ca-cert\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-envoy\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864938 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864765 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-credential-socket\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.864982 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-certs\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.865020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.864960 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-data\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.865264 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.865240 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-socket\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.865438 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.865422 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istiod-ca-cert\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.867074 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.867053 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-envoy\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.867217 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.867199 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-podinfo\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.873841 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.873820 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzj6k\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-kube-api-access-hzj6k\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:02.875073 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:02.875052 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-token\") pod \"e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:03.005522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:03.005484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:03.147301 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:03.147278 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh"] Jul 07 19:34:03.148680 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:34:03.148654 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412f1f61_732f_4f3e_b72e_aa9ee90b7752.slice/crio-fbe5fa568a21bffa9d5be5e20c08624470df03f578333526df04e9ad64eff76d WatchSource:0}: Error finding container fbe5fa568a21bffa9d5be5e20c08624470df03f578333526df04e9ad64eff76d: Status 404 returned error can't find the container with id fbe5fa568a21bffa9d5be5e20c08624470df03f578333526df04e9ad64eff76d Jul 07 19:34:03.150990 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:03.150956 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:34:03.151091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:03.151027 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:34:03.151091 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:03.151063 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:34:03.400963 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:03.400850 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" event={"ID":"412f1f61-732f-4f3e-b72e-aa9ee90b7752","Type":"ContainerStarted","Data":"22c381d1f3022e577d4298dfa9febf9b2b2de79b7bcf4ac60f2f5acfc3a2d6b2"} Jul 07 19:34:03.400963 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:03.400948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" event={"ID":"412f1f61-732f-4f3e-b72e-aa9ee90b7752","Type":"ContainerStarted","Data":"fbe5fa568a21bffa9d5be5e20c08624470df03f578333526df04e9ad64eff76d"} Jul 07 19:34:03.437221 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:03.437170 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" podStartSLOduration=1.437154125 podStartE2EDuration="1.437154125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:03.435234742 +0000 UTC m=+1549.370860103" watchObservedRunningTime="2026-07-07 19:34:03.437154125 +0000 UTC m=+1549.372779485" Jul 07 19:34:03.907940 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:03.907906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:04.006057 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.006019 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:04.008047 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.008020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:04.011142 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.011122 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:04.164945 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.164846 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25"] Jul 07 19:34:04.169338 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.169301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.173346 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.173318 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-8b7b410f-serving-cert\"" Jul 07 19:34:04.181431 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.181409 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25"] Jul 07 19:34:04.210106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.210057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:04.278347 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.278317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f18240-d637-4051-a9f6-d3bdee45702c-openshift-service-ca\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.278521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.278411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6mmd\" (UniqueName: \"kubernetes.io/projected/97f18240-d637-4051-a9f6-d3bdee45702c-kube-api-access-t6mmd\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.278521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.278468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f18240-d637-4051-a9f6-d3bdee45702c-maas-api-tls\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.379834 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.379796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f18240-d637-4051-a9f6-d3bdee45702c-openshift-service-ca\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.380055 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.379917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6mmd\" (UniqueName: \"kubernetes.io/projected/97f18240-d637-4051-a9f6-d3bdee45702c-kube-api-access-t6mmd\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.380055 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.379984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f18240-d637-4051-a9f6-d3bdee45702c-maas-api-tls\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.380180 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:04.380134 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-8b7b410f-serving-cert: secret "maas-api-e2e-ait-8b7b410f-serving-cert" not found Jul 07 19:34:04.380233 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:04.380198 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f18240-d637-4051-a9f6-d3bdee45702c-maas-api-tls podName:97f18240-d637-4051-a9f6-d3bdee45702c nodeName:}" failed. No retries permitted until 2026-07-07 19:34:04.880176256 +0000 UTC m=+1550.815801610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97f18240-d637-4051-a9f6-d3bdee45702c-maas-api-tls") pod "maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" (UID: "97f18240-d637-4051-a9f6-d3bdee45702c") : secret "maas-api-e2e-ait-8b7b410f-serving-cert" not found Jul 07 19:34:04.380568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.380547 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f18240-d637-4051-a9f6-d3bdee45702c-openshift-service-ca\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.392263 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.392242 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6mmd\" (UniqueName: \"kubernetes.io/projected/97f18240-d637-4051-a9f6-d3bdee45702c-kube-api-access-t6mmd\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.406350 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.406326 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:04.407240 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.407218 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:04.592217 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.592178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:04.745236 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.745198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:04.845645 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.845610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:04.885860 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.885825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f18240-d637-4051-a9f6-d3bdee45702c-maas-api-tls\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:04.889303 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:04.889276 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f18240-d637-4051-a9f6-d3bdee45702c-maas-api-tls\") pod \"maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:05.083992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:05.083898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:05.219104 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:05.219071 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25"] Jul 07 19:34:05.220968 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:34:05.220935 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f18240_d637_4051_a9f6_d3bdee45702c.slice/crio-9190f6688e6d3a8a9d765e7bf461602a3187a4cba50d1167f2313d461e2a472f WatchSource:0}: Error finding container 9190f6688e6d3a8a9d765e7bf461602a3187a4cba50d1167f2313d461e2a472f: Status 404 returned error can't find the container with id 9190f6688e6d3a8a9d765e7bf461602a3187a4cba50d1167f2313d461e2a472f Jul 07 19:34:05.222831 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:05.222813 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:34:05.412147 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:05.412060 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" event={"ID":"97f18240-d637-4051-a9f6-d3bdee45702c","Type":"ContainerStarted","Data":"9190f6688e6d3a8a9d765e7bf461602a3187a4cba50d1167f2313d461e2a472f"} Jul 07 19:34:06.375381 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:06.375352 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:06.418476 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:06.418438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" event={"ID":"97f18240-d637-4051-a9f6-d3bdee45702c","Type":"ContainerStarted","Data":"815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4"} Jul 07 19:34:06.418667 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:06.418524 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:06.437496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:06.437444 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" podStartSLOduration=1.9870042460000001 podStartE2EDuration="2.437425336s" totalImagesPullingTime="450.42109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:06.436430057 +0000 UTC m=+1552.372055418" watchObservedRunningTime="2026-07-07 19:34:06.437425336 +0000 UTC m=+1552.373050695" Jul 07 19:34:08.957016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:08.956979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:09.126429 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.126395 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh"] Jul 07 19:34:09.126605 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.126573 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" podUID="412f1f61-732f-4f3e-b72e-aa9ee90b7752" containerName="istio-proxy" containerID="cri-o://22c381d1f3022e577d4298dfa9febf9b2b2de79b7bcf4ac60f2f5acfc3a2d6b2" gracePeriod=30 Jul 07 19:34:09.153927 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.153892 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:09.251232 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.251195 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:09.367640 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.367600 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25"] Jul 07 19:34:09.367960 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.367822 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" podUID="97f18240-d637-4051-a9f6-d3bdee45702c" containerName="maas-api" containerID="cri-o://815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4" gracePeriod=30 Jul 07 19:34:09.399949 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.399918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:09.675267 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.675239 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:09.835857 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.835778 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f18240-d637-4051-a9f6-d3bdee45702c-maas-api-tls\") pod \"97f18240-d637-4051-a9f6-d3bdee45702c\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " Jul 07 19:34:09.835857 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.835832 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f18240-d637-4051-a9f6-d3bdee45702c-openshift-service-ca\") pod \"97f18240-d637-4051-a9f6-d3bdee45702c\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " Jul 07 19:34:09.835857 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.835852 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6mmd\" (UniqueName: \"kubernetes.io/projected/97f18240-d637-4051-a9f6-d3bdee45702c-kube-api-access-t6mmd\") pod \"97f18240-d637-4051-a9f6-d3bdee45702c\" (UID: \"97f18240-d637-4051-a9f6-d3bdee45702c\") " Jul 07 19:34:09.836247 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.836218 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f18240-d637-4051-a9f6-d3bdee45702c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97f18240-d637-4051-a9f6-d3bdee45702c" (UID: "97f18240-d637-4051-a9f6-d3bdee45702c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:34:09.838067 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.838043 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f18240-d637-4051-a9f6-d3bdee45702c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97f18240-d637-4051-a9f6-d3bdee45702c" (UID: "97f18240-d637-4051-a9f6-d3bdee45702c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:34:09.838150 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.838120 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f18240-d637-4051-a9f6-d3bdee45702c-kube-api-access-t6mmd" (OuterVolumeSpecName: "kube-api-access-t6mmd") pod "97f18240-d637-4051-a9f6-d3bdee45702c" (UID: "97f18240-d637-4051-a9f6-d3bdee45702c"). InnerVolumeSpecName "kube-api-access-t6mmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:09.937391 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.937354 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f18240-d637-4051-a9f6-d3bdee45702c-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:09.937391 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.937386 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6mmd\" (UniqueName: \"kubernetes.io/projected/97f18240-d637-4051-a9f6-d3bdee45702c-kube-api-access-t6mmd\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:09.937391 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:09.937397 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f18240-d637-4051-a9f6-d3bdee45702c-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:10.441524 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.441489 2577 generic.go:356] "Generic (PLEG): container finished" podID="97f18240-d637-4051-a9f6-d3bdee45702c" containerID="815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4" exitCode=0 Jul 07 19:34:10.441934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.441587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" event={"ID":"97f18240-d637-4051-a9f6-d3bdee45702c","Type":"ContainerDied","Data":"815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4"} Jul 07 19:34:10.441934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.441656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" event={"ID":"97f18240-d637-4051-a9f6-d3bdee45702c","Type":"ContainerDied","Data":"9190f6688e6d3a8a9d765e7bf461602a3187a4cba50d1167f2313d461e2a472f"} Jul 07 19:34:10.441934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.441680 2577 scope.go:117] "RemoveContainer" containerID="815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4" Jul 07 19:34:10.441934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.441789 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25" Jul 07 19:34:10.451838 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.451820 2577 scope.go:117] "RemoveContainer" containerID="815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4" Jul 07 19:34:10.452125 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:10.452105 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4\": container with ID starting with 815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4 not found: ID does not exist" containerID="815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4" Jul 07 19:34:10.452174 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.452136 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4"} err="failed to get container status \"815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4\": rpc error: code = NotFound desc = could not find container \"815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4\": container with ID starting with 815a949b8a09f82221f0106351a368a6abb1a9a665aca3981c54551397914be4 not found: ID does not exist" Jul 07 19:34:10.466305 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.466282 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25"] Jul 07 19:34:10.473240 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.473215 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-8b7b410f-94c9ffcb8-p4d25"] Jul 07 19:34:10.608541 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:10.608495 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f18240-d637-4051-a9f6-d3bdee45702c" path="/var/lib/kubelet/pods/97f18240-d637-4051-a9f6-d3bdee45702c/volumes" Jul 07 19:34:11.211744 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:11.211696 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:15.053561 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.053504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:15.114979 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.114947 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:15.233282 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.233226 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6"] Jul 07 19:34:15.234010 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.233991 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f18240-d637-4051-a9f6-d3bdee45702c" containerName="maas-api" Jul 07 19:34:15.234120 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.234012 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f18240-d637-4051-a9f6-d3bdee45702c" containerName="maas-api" Jul 07 19:34:15.234120 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.234106 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f18240-d637-4051-a9f6-d3bdee45702c" containerName="maas-api" Jul 07 19:34:15.249615 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.249588 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6"] Jul 07 19:34:15.249727 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.249689 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.252295 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.252277 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6517f0a0-openshift-default-dockercfg-rdl9b\"" Jul 07 19:34:15.275269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.275238 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:15.375831 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.375798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:15.392392 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.392369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-envoy\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.392498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.392414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-credential-socket\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.392498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.392460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-data\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.392498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.392486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab33651-feba-4120-b85f-235c2aa78b4c-istio-podinfo\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.392603 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.392506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-socket\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.392603 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.392527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rlg\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-kube-api-access-k6rlg\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.392673 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.392603 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-certs\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.392673 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.392632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-istio-token\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.392673 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.392654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab33651-feba-4120-b85f-235c2aa78b4c-istiod-ca-cert\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.470451 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.470421 2577 generic.go:356] "Generic (PLEG): container finished" podID="412f1f61-732f-4f3e-b72e-aa9ee90b7752" containerID="22c381d1f3022e577d4298dfa9febf9b2b2de79b7bcf4ac60f2f5acfc3a2d6b2" exitCode=0 Jul 07 19:34:15.470600 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.470465 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" event={"ID":"412f1f61-732f-4f3e-b72e-aa9ee90b7752","Type":"ContainerDied","Data":"22c381d1f3022e577d4298dfa9febf9b2b2de79b7bcf4ac60f2f5acfc3a2d6b2"} Jul 07 19:34:15.493606 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.493574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-certs\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.493830 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.493808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-istio-token\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494034 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab33651-feba-4120-b85f-235c2aa78b4c-istiod-ca-cert\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494190 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-envoy\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494033 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-certs\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-credential-socket\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-data\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab33651-feba-4120-b85f-235c2aa78b4c-istio-podinfo\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-socket\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rlg\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-kube-api-access-k6rlg\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494674 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-credential-socket\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494699 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab33651-feba-4120-b85f-235c2aa78b4c-istiod-ca-cert\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494863 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-socket\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.494997 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.494959 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-data\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.496827 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.496808 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-envoy\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.497565 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.497522 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab33651-feba-4120-b85f-235c2aa78b4c-istio-podinfo\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.502910 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.502864 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rlg\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-kube-api-access-k6rlg\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.503241 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.503220 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-istio-token\") pod \"e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.561075 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.561010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:15.589863 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.589824 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:15.697397 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697366 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-credential-socket\") pod \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " Jul 07 19:34:15.697575 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697413 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-socket\") pod \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " Jul 07 19:34:15.697575 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697470 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istiod-ca-cert\") pod \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " Jul 07 19:34:15.697575 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697517 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-data\") pod \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " Jul 07 19:34:15.697575 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697567 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-certs\") pod \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " Jul 07 19:34:15.697793 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697618 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-token\") pod \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " Jul 07 19:34:15.697793 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697626 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "412f1f61-732f-4f3e-b72e-aa9ee90b7752" (UID: "412f1f61-732f-4f3e-b72e-aa9ee90b7752"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:15.697793 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697649 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-podinfo\") pod \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " Jul 07 19:34:15.697793 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697728 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzj6k\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-kube-api-access-hzj6k\") pod \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " Jul 07 19:34:15.697793 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697761 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "412f1f61-732f-4f3e-b72e-aa9ee90b7752" (UID: "412f1f61-732f-4f3e-b72e-aa9ee90b7752"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:15.698131 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697848 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "412f1f61-732f-4f3e-b72e-aa9ee90b7752" (UID: "412f1f61-732f-4f3e-b72e-aa9ee90b7752"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:34:15.698131 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.697913 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-envoy\") pod \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\" (UID: \"412f1f61-732f-4f3e-b72e-aa9ee90b7752\") " Jul 07 19:34:15.698131 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.698102 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "412f1f61-732f-4f3e-b72e-aa9ee90b7752" (UID: "412f1f61-732f-4f3e-b72e-aa9ee90b7752"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:15.698453 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.698382 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istiod-ca-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:15.698453 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.698400 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-certs\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:15.698453 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.698412 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-credential-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:15.698453 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.698425 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-workload-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:15.698657 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.698566 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-data" (OuterVolumeSpecName: "istio-data") pod "412f1f61-732f-4f3e-b72e-aa9ee90b7752" (UID: "412f1f61-732f-4f3e-b72e-aa9ee90b7752"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:15.700053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.700025 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-token" (OuterVolumeSpecName: "istio-token") pod "412f1f61-732f-4f3e-b72e-aa9ee90b7752" (UID: "412f1f61-732f-4f3e-b72e-aa9ee90b7752"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:15.700053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.700028 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "412f1f61-732f-4f3e-b72e-aa9ee90b7752" (UID: "412f1f61-732f-4f3e-b72e-aa9ee90b7752"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:34:15.700598 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.700570 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "412f1f61-732f-4f3e-b72e-aa9ee90b7752" (UID: "412f1f61-732f-4f3e-b72e-aa9ee90b7752"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:15.700659 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.700580 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-kube-api-access-hzj6k" (OuterVolumeSpecName: "kube-api-access-hzj6k") pod "412f1f61-732f-4f3e-b72e-aa9ee90b7752" (UID: "412f1f61-732f-4f3e-b72e-aa9ee90b7752"). InnerVolumeSpecName "kube-api-access-hzj6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:15.712998 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.712921 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6"] Jul 07 19:34:15.715386 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:34:15.715365 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab33651_feba_4120_b85f_235c2aa78b4c.slice/crio-ad3cad3496086440aa3e4f0f72795e9c0e91fcdbf284756c7c7682741c097388 WatchSource:0}: Error finding container ad3cad3496086440aa3e4f0f72795e9c0e91fcdbf284756c7c7682741c097388: Status 404 returned error can't find the container with id ad3cad3496086440aa3e4f0f72795e9c0e91fcdbf284756c7c7682741c097388 Jul 07 19:34:15.717569 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.717539 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:34:15.717699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.717604 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:34:15.717699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.717629 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:34:15.799212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.799180 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-data\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:15.799212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.799202 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-token\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:15.799212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.799210 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-podinfo\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:15.799212 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.799218 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzj6k\" (UniqueName: \"kubernetes.io/projected/412f1f61-732f-4f3e-b72e-aa9ee90b7752-kube-api-access-hzj6k\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:15.799525 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:15.799227 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412f1f61-732f-4f3e-b72e-aa9ee90b7752-istio-envoy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:16.454820 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.454788 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:16.479689 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.478700 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" Jul 07 19:34:16.479689 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.478715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh" event={"ID":"412f1f61-732f-4f3e-b72e-aa9ee90b7752","Type":"ContainerDied","Data":"fbe5fa568a21bffa9d5be5e20c08624470df03f578333526df04e9ad64eff76d"} Jul 07 19:34:16.479689 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.478780 2577 scope.go:117] "RemoveContainer" containerID="22c381d1f3022e577d4298dfa9febf9b2b2de79b7bcf4ac60f2f5acfc3a2d6b2" Jul 07 19:34:16.481247 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.481217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" event={"ID":"dab33651-feba-4120-b85f-235c2aa78b4c","Type":"ContainerStarted","Data":"cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f"} Jul 07 19:34:16.481371 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.481273 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" event={"ID":"dab33651-feba-4120-b85f-235c2aa78b4c","Type":"ContainerStarted","Data":"ad3cad3496086440aa3e4f0f72795e9c0e91fcdbf284756c7c7682741c097388"} Jul 07 19:34:16.508464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.508419 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" podStartSLOduration=1.508404146 podStartE2EDuration="1.508404146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:16.506241241 +0000 UTC m=+1562.441866603" watchObservedRunningTime="2026-07-07 19:34:16.508404146 +0000 UTC m=+1562.444029505" Jul 07 19:34:16.526758 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.526732 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh"] Jul 07 19:34:16.531377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.531348 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z"] Jul 07 19:34:16.531796 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.531783 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412f1f61-732f-4f3e-b72e-aa9ee90b7752" containerName="istio-proxy" Jul 07 19:34:16.531796 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.531797 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="412f1f61-732f-4f3e-b72e-aa9ee90b7752" containerName="istio-proxy" Jul 07 19:34:16.531992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.531951 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="412f1f61-732f-4f3e-b72e-aa9ee90b7752" containerName="istio-proxy" Jul 07 19:34:16.535115 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.535074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:16.537491 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.537455 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8b7b410f-openshift-default-5cffc6647f-b2sgh"] Jul 07 19:34:16.537588 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.537562 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-6517f0a0-serving-cert\"" Jul 07 19:34:16.540207 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.540162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:16.546249 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.545981 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z"] Jul 07 19:34:16.561499 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.561468 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:16.567263 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.567239 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:16.604181 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.604151 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="412f1f61-732f-4f3e-b72e-aa9ee90b7752" path="/var/lib/kubelet/pods/412f1f61-732f-4f3e-b72e-aa9ee90b7752/volumes" Jul 07 19:34:16.612058 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.612029 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:16.708377 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.708297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqj7g\" (UniqueName: \"kubernetes.io/projected/e771e6e3-9e2c-4a30-af72-96e4bc069cce-kube-api-access-fqj7g\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:16.708531 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.708436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e771e6e3-9e2c-4a30-af72-96e4bc069cce-maas-api-tls\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:16.708531 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.708497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e771e6e3-9e2c-4a30-af72-96e4bc069cce-openshift-service-ca\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:16.809095 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.809062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e771e6e3-9e2c-4a30-af72-96e4bc069cce-maas-api-tls\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:16.809299 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.809123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e771e6e3-9e2c-4a30-af72-96e4bc069cce-openshift-service-ca\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:16.809299 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.809168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqj7g\" (UniqueName: \"kubernetes.io/projected/e771e6e3-9e2c-4a30-af72-96e4bc069cce-kube-api-access-fqj7g\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:16.809299 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:16.809228 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-6517f0a0-serving-cert: secret "maas-api-e2e-ait-6517f0a0-serving-cert" not found Jul 07 19:34:16.809474 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:16.809300 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e771e6e3-9e2c-4a30-af72-96e4bc069cce-maas-api-tls podName:e771e6e3-9e2c-4a30-af72-96e4bc069cce nodeName:}" failed. No retries permitted until 2026-07-07 19:34:17.30927844 +0000 UTC m=+1563.244903798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e771e6e3-9e2c-4a30-af72-96e4bc069cce-maas-api-tls") pod "maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" (UID: "e771e6e3-9e2c-4a30-af72-96e4bc069cce") : secret "maas-api-e2e-ait-6517f0a0-serving-cert" not found Jul 07 19:34:16.809826 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.809802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e771e6e3-9e2c-4a30-af72-96e4bc069cce-openshift-service-ca\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:16.819791 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.819762 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqj7g\" (UniqueName: \"kubernetes.io/projected/e771e6e3-9e2c-4a30-af72-96e4bc069cce-kube-api-access-fqj7g\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:16.911461 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:16.911426 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:17.246136 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:17.246099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:17.314356 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:17.314323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e771e6e3-9e2c-4a30-af72-96e4bc069cce-maas-api-tls\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:17.316922 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:17.316869 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e771e6e3-9e2c-4a30-af72-96e4bc069cce-maas-api-tls\") pod \"maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:17.435085 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:17.435043 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:17.447781 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:17.447751 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:17.489350 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:17.489323 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:17.490375 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:17.490353 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:17.644427 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:17.644403 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z"] Jul 07 19:34:17.646124 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:34:17.646099 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode771e6e3_9e2c_4a30_af72_96e4bc069cce.slice/crio-93bc7e5ccc7e988e4ce03da954b15dd90d97581751c8769cbb04fadf7c90ad2c WatchSource:0}: Error finding container 93bc7e5ccc7e988e4ce03da954b15dd90d97581751c8769cbb04fadf7c90ad2c: Status 404 returned error can't find the container with id 93bc7e5ccc7e988e4ce03da954b15dd90d97581751c8769cbb04fadf7c90ad2c Jul 07 19:34:18.468432 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:18.468396 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:18.497393 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:18.497359 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" event={"ID":"e771e6e3-9e2c-4a30-af72-96e4bc069cce","Type":"ContainerStarted","Data":"2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409"} Jul 07 19:34:18.497845 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:18.497433 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" event={"ID":"e771e6e3-9e2c-4a30-af72-96e4bc069cce","Type":"ContainerStarted","Data":"93bc7e5ccc7e988e4ce03da954b15dd90d97581751c8769cbb04fadf7c90ad2c"} Jul 07 19:34:18.521652 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:18.521605 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" podStartSLOduration=2.15889337 podStartE2EDuration="2.521590923s" totalImagesPullingTime="362.697553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:18.519930469 +0000 UTC m=+1564.455555831" watchObservedRunningTime="2026-07-07 19:34:18.521590923 +0000 UTC m=+1564.457216283" Jul 07 19:34:19.503618 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:19.503577 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:21.560711 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:21.560675 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:22.072982 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.072955 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z"] Jul 07 19:34:22.073162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.073137 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" podUID="e771e6e3-9e2c-4a30-af72-96e4bc069cce" containerName="maas-api" containerID="cri-o://2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409" gracePeriod=30 Jul 07 19:34:22.111056 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.110966 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:22.360743 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.360716 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:22.463544 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.463508 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqj7g\" (UniqueName: \"kubernetes.io/projected/e771e6e3-9e2c-4a30-af72-96e4bc069cce-kube-api-access-fqj7g\") pod \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " Jul 07 19:34:22.463688 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.463552 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e771e6e3-9e2c-4a30-af72-96e4bc069cce-maas-api-tls\") pod \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " Jul 07 19:34:22.463688 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.463676 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e771e6e3-9e2c-4a30-af72-96e4bc069cce-openshift-service-ca\") pod \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\" (UID: \"e771e6e3-9e2c-4a30-af72-96e4bc069cce\") " Jul 07 19:34:22.464001 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.463978 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e771e6e3-9e2c-4a30-af72-96e4bc069cce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e771e6e3-9e2c-4a30-af72-96e4bc069cce" (UID: "e771e6e3-9e2c-4a30-af72-96e4bc069cce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:34:22.464084 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.464057 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e771e6e3-9e2c-4a30-af72-96e4bc069cce-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:22.465780 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.465758 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e771e6e3-9e2c-4a30-af72-96e4bc069cce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e771e6e3-9e2c-4a30-af72-96e4bc069cce" (UID: "e771e6e3-9e2c-4a30-af72-96e4bc069cce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:34:22.465889 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.465814 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e771e6e3-9e2c-4a30-af72-96e4bc069cce-kube-api-access-fqj7g" (OuterVolumeSpecName: "kube-api-access-fqj7g") pod "e771e6e3-9e2c-4a30-af72-96e4bc069cce" (UID: "e771e6e3-9e2c-4a30-af72-96e4bc069cce"). InnerVolumeSpecName "kube-api-access-fqj7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:22.523832 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.523796 2577 generic.go:356] "Generic (PLEG): container finished" podID="e771e6e3-9e2c-4a30-af72-96e4bc069cce" containerID="2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409" exitCode=0 Jul 07 19:34:22.524022 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.523863 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" Jul 07 19:34:22.524022 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.523903 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" event={"ID":"e771e6e3-9e2c-4a30-af72-96e4bc069cce","Type":"ContainerDied","Data":"2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409"} Jul 07 19:34:22.524022 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.523972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z" event={"ID":"e771e6e3-9e2c-4a30-af72-96e4bc069cce","Type":"ContainerDied","Data":"93bc7e5ccc7e988e4ce03da954b15dd90d97581751c8769cbb04fadf7c90ad2c"} Jul 07 19:34:22.524022 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.524006 2577 scope.go:117] "RemoveContainer" containerID="2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409" Jul 07 19:34:22.533767 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.533750 2577 scope.go:117] "RemoveContainer" containerID="2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409" Jul 07 19:34:22.534030 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:22.534011 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409\": container with ID starting with 2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409 not found: ID does not exist" containerID="2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409" Jul 07 19:34:22.534077 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.534039 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409"} err="failed to get container status \"2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409\": rpc error: code = NotFound desc = could not find container \"2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409\": container with ID starting with 2a35620c9750e09c036d411f009145fe1d7005b8363fa910112aab2edfe95409 not found: ID does not exist" Jul 07 19:34:22.545978 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.545955 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z"] Jul 07 19:34:22.549635 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.549615 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-6517f0a0-5fb9b846d4-6v79z"] Jul 07 19:34:22.565221 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.565202 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqj7g\" (UniqueName: \"kubernetes.io/projected/e771e6e3-9e2c-4a30-af72-96e4bc069cce-kube-api-access-fqj7g\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:22.565500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.565225 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e771e6e3-9e2c-4a30-af72-96e4bc069cce-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:22.608565 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:22.608474 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e771e6e3-9e2c-4a30-af72-96e4bc069cce" path="/var/lib/kubelet/pods/e771e6e3-9e2c-4a30-af72-96e4bc069cce/volumes" Jul 07 19:34:27.676711 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:27.676674 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6"] Jul 07 19:34:27.677202 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:27.676962 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" podUID="dab33651-feba-4120-b85f-235c2aa78b4c" containerName="istio-proxy" containerID="cri-o://cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f" gracePeriod=30 Jul 07 19:34:27.697151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:27.697096 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:27.794366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:27.794320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:28.235633 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.235609 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:28.327637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.327609 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab33651-feba-4120-b85f-235c2aa78b4c-istiod-ca-cert\") pod \"dab33651-feba-4120-b85f-235c2aa78b4c\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " Jul 07 19:34:28.327772 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.327673 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-envoy\") pod \"dab33651-feba-4120-b85f-235c2aa78b4c\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " Jul 07 19:34:28.327772 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.327702 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-data\") pod \"dab33651-feba-4120-b85f-235c2aa78b4c\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " Jul 07 19:34:28.327772 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.327725 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-istio-token\") pod \"dab33651-feba-4120-b85f-235c2aa78b4c\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " Jul 07 19:34:28.327772 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.327745 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-socket\") pod \"dab33651-feba-4120-b85f-235c2aa78b4c\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " Jul 07 19:34:28.327772 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.327762 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6rlg\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-kube-api-access-k6rlg\") pod \"dab33651-feba-4120-b85f-235c2aa78b4c\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " Jul 07 19:34:28.328034 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.327785 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-certs\") pod \"dab33651-feba-4120-b85f-235c2aa78b4c\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " Jul 07 19:34:28.328034 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.327820 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab33651-feba-4120-b85f-235c2aa78b4c-istio-podinfo\") pod \"dab33651-feba-4120-b85f-235c2aa78b4c\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " Jul 07 19:34:28.328034 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.327867 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-credential-socket\") pod \"dab33651-feba-4120-b85f-235c2aa78b4c\" (UID: \"dab33651-feba-4120-b85f-235c2aa78b4c\") " Jul 07 19:34:28.328184 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.328041 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab33651-feba-4120-b85f-235c2aa78b4c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dab33651-feba-4120-b85f-235c2aa78b4c" (UID: "dab33651-feba-4120-b85f-235c2aa78b4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:34:28.328184 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.328113 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dab33651-feba-4120-b85f-235c2aa78b4c" (UID: "dab33651-feba-4120-b85f-235c2aa78b4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:28.328305 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.328231 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dab33651-feba-4120-b85f-235c2aa78b4c" (UID: "dab33651-feba-4120-b85f-235c2aa78b4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:28.328358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.328311 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab33651-feba-4120-b85f-235c2aa78b4c-istiod-ca-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:28.328358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.328331 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-certs\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:28.328358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.328346 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-credential-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:28.328668 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.328487 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-data" (OuterVolumeSpecName: "istio-data") pod "dab33651-feba-4120-b85f-235c2aa78b4c" (UID: "dab33651-feba-4120-b85f-235c2aa78b4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:28.328730 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.328693 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dab33651-feba-4120-b85f-235c2aa78b4c" (UID: "dab33651-feba-4120-b85f-235c2aa78b4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:28.330181 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.330159 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dab33651-feba-4120-b85f-235c2aa78b4c" (UID: "dab33651-feba-4120-b85f-235c2aa78b4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:28.330306 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.330185 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-istio-token" (OuterVolumeSpecName: "istio-token") pod "dab33651-feba-4120-b85f-235c2aa78b4c" (UID: "dab33651-feba-4120-b85f-235c2aa78b4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:28.330306 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.330255 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-kube-api-access-k6rlg" (OuterVolumeSpecName: "kube-api-access-k6rlg") pod "dab33651-feba-4120-b85f-235c2aa78b4c" (UID: "dab33651-feba-4120-b85f-235c2aa78b4c"). InnerVolumeSpecName "kube-api-access-k6rlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:28.330468 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.330450 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dab33651-feba-4120-b85f-235c2aa78b4c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dab33651-feba-4120-b85f-235c2aa78b4c" (UID: "dab33651-feba-4120-b85f-235c2aa78b4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:34:28.428846 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.428816 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-envoy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:28.428846 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.428841 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-istio-data\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:28.428846 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.428849 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-istio-token\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:28.429070 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.428857 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab33651-feba-4120-b85f-235c2aa78b4c-workload-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:28.429070 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.428866 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rlg\" (UniqueName: \"kubernetes.io/projected/dab33651-feba-4120-b85f-235c2aa78b4c-kube-api-access-k6rlg\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:28.429070 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.428901 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab33651-feba-4120-b85f-235c2aa78b4c-istio-podinfo\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:28.561228 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.561139 2577 generic.go:356] "Generic (PLEG): container finished" podID="dab33651-feba-4120-b85f-235c2aa78b4c" containerID="cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f" exitCode=0 Jul 07 19:34:28.561228 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.561186 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" Jul 07 19:34:28.561470 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.561229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" event={"ID":"dab33651-feba-4120-b85f-235c2aa78b4c","Type":"ContainerDied","Data":"cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f"} Jul 07 19:34:28.561470 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.561289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6" event={"ID":"dab33651-feba-4120-b85f-235c2aa78b4c","Type":"ContainerDied","Data":"ad3cad3496086440aa3e4f0f72795e9c0e91fcdbf284756c7c7682741c097388"} Jul 07 19:34:28.561470 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.561326 2577 scope.go:117] "RemoveContainer" containerID="cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f" Jul 07 19:34:28.571008 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.570991 2577 scope.go:117] "RemoveContainer" containerID="cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f" Jul 07 19:34:28.571241 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:28.571225 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f\": container with ID starting with cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f not found: ID does not exist" containerID="cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f" Jul 07 19:34:28.571292 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.571247 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f"} err="failed to get container status \"cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f\": rpc error: code = NotFound desc = could not find container \"cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f\": container with ID starting with cf1fdefada665c6a286d1ba5a1c38435ae02644d773fde3699a66bfc41edd34f not found: ID does not exist" Jul 07 19:34:28.585906 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.585862 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6"] Jul 07 19:34:28.589067 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.589048 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6517f0a0-openshift-default-6469bb679f-c9qb6"] Jul 07 19:34:28.610249 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:28.610225 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab33651-feba-4120-b85f-235c2aa78b4c" path="/var/lib/kubelet/pods/dab33651-feba-4120-b85f-235c2aa78b4c/volumes" Jul 07 19:34:29.700086 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:29.700049 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:33.771840 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.771798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:33.872529 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.872486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:33.939972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.939939 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6"] Jul 07 19:34:33.940686 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.940663 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e771e6e3-9e2c-4a30-af72-96e4bc069cce" containerName="maas-api" Jul 07 19:34:33.940686 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.940686 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e771e6e3-9e2c-4a30-af72-96e4bc069cce" containerName="maas-api" Jul 07 19:34:33.940854 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.940703 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab33651-feba-4120-b85f-235c2aa78b4c" containerName="istio-proxy" Jul 07 19:34:33.940854 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.940709 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab33651-feba-4120-b85f-235c2aa78b4c" containerName="istio-proxy" Jul 07 19:34:33.940854 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.940808 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab33651-feba-4120-b85f-235c2aa78b4c" containerName="istio-proxy" Jul 07 19:34:33.940854 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.940820 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e771e6e3-9e2c-4a30-af72-96e4bc069cce" containerName="maas-api" Jul 07 19:34:33.944366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.944349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:33.947083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.947062 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-edcb1097-openshift-default-dockercfg-q7pf2\"" Jul 07 19:34:33.958412 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.958386 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6"] Jul 07 19:34:33.987766 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:33.987726 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:34.078666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.078568 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:34.081811 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.081785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-socket\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.081959 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.081818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-podinfo\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.081959 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.081842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-certs\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.081959 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.081941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm796\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-kube-api-access-hm796\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.082124 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.081988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istiod-ca-cert\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.082124 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.082040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-token\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.082124 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.082067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-envoy\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.082261 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.082136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-credential-socket\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.082261 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.082163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-data\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.182990 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.182949 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istiod-ca-cert\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-token\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-envoy\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-credential-socket\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-data\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183367 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-socket\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183367 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-podinfo\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183367 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-certs\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183367 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm796\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-kube-api-access-hm796\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183560 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183433 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-credential-socket\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.183776 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.183754 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istiod-ca-cert\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.184054 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.184033 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-socket\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.184147 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.184072 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-certs\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.184202 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.184161 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-data\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.185999 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.185975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-envoy\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.186490 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.186473 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-podinfo\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.192411 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.192386 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-token\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.192500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.192475 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm796\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-kube-api-access-hm796\") pod \"e2e-derive-edcb1097-openshift-default-5d55776787-npvx6\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.255498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.255465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:34.391825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.391802 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6"] Jul 07 19:34:34.393784 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:34:34.393750 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9924c5a8_0642_428c_ac9d_dd17b0acb5c4.slice/crio-5e4665eb1142206d0424030006195b49fe0acb4b7f46ce8b98a8d8df97b8a667 WatchSource:0}: Error finding container 5e4665eb1142206d0424030006195b49fe0acb4b7f46ce8b98a8d8df97b8a667: Status 404 returned error can't find the container with id 5e4665eb1142206d0424030006195b49fe0acb4b7f46ce8b98a8d8df97b8a667 Jul 07 19:34:34.396118 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.396082 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:34:34.396191 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.396170 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:34:34.396241 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.396217 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:34:34.597013 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.596919 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" event={"ID":"9924c5a8-0642-428c-ac9d-dd17b0acb5c4","Type":"ContainerStarted","Data":"9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17"} Jul 07 19:34:34.597013 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.596988 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" event={"ID":"9924c5a8-0642-428c-ac9d-dd17b0acb5c4","Type":"ContainerStarted","Data":"5e4665eb1142206d0424030006195b49fe0acb4b7f46ce8b98a8d8df97b8a667"} Jul 07 19:34:34.622991 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:34.622936 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" podStartSLOduration=1.62291765 podStartE2EDuration="1.62291765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:34.619953099 +0000 UTC m=+1580.555578459" watchObservedRunningTime="2026-07-07 19:34:34.62291765 +0000 UTC m=+1580.558543011" Jul 07 19:34:35.161579 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.161539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.244083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.244054 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9"] Jul 07 19:34:35.246958 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.246929 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.247344 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.247248 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.249749 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.249728 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-edcb1097-serving-cert\"" Jul 07 19:34:35.255618 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.255598 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:35.257699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.257677 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9"] Jul 07 19:34:35.260806 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.260785 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:35.295554 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.295525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2317815-5702-4343-9ecf-91d3684b5ffa-openshift-service-ca\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.295699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.295675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dflc\" (UniqueName: \"kubernetes.io/projected/e2317815-5702-4343-9ecf-91d3684b5ffa-kube-api-access-9dflc\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.295801 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.295714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2317815-5702-4343-9ecf-91d3684b5ffa-maas-api-tls\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.317626 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.317597 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.396351 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.396316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dflc\" (UniqueName: \"kubernetes.io/projected/e2317815-5702-4343-9ecf-91d3684b5ffa-kube-api-access-9dflc\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.396538 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.396360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2317815-5702-4343-9ecf-91d3684b5ffa-maas-api-tls\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.396538 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:35.396479 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-edcb1097-serving-cert: secret "maas-api-e2e-derive-edcb1097-serving-cert" not found Jul 07 19:34:35.396538 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:35.396538 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2317815-5702-4343-9ecf-91d3684b5ffa-maas-api-tls podName:e2317815-5702-4343-9ecf-91d3684b5ffa nodeName:}" failed. No retries permitted until 2026-07-07 19:34:35.896517428 +0000 UTC m=+1581.832142782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2317815-5702-4343-9ecf-91d3684b5ffa-maas-api-tls") pod "maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" (UID: "e2317815-5702-4343-9ecf-91d3684b5ffa") : secret "maas-api-e2e-derive-edcb1097-serving-cert" not found Jul 07 19:34:35.396719 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.396568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2317815-5702-4343-9ecf-91d3684b5ffa-openshift-service-ca\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.397195 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.397176 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2317815-5702-4343-9ecf-91d3684b5ffa-openshift-service-ca\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.406122 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.406095 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dflc\" (UniqueName: \"kubernetes.io/projected/e2317815-5702-4343-9ecf-91d3684b5ffa-kube-api-access-9dflc\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.602655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.602626 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:35.603448 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.603431 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:35.743049 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.743014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.842452 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.842421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.902027 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.901943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2317815-5702-4343-9ecf-91d3684b5ffa-maas-api-tls\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:35.904435 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:35.904417 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2317815-5702-4343-9ecf-91d3684b5ffa-maas-api-tls\") pod \"maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:36.159502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:36.159417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:36.509670 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:36.509485 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9"] Jul 07 19:34:36.514236 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:34:36.514204 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2317815_5702_4343_9ecf_91d3684b5ffa.slice/crio-5ee6447607ee70e390d79d38ec16f7110cc30404389b2a5427bd077dced34c85 WatchSource:0}: Error finding container 5ee6447607ee70e390d79d38ec16f7110cc30404389b2a5427bd077dced34c85: Status 404 returned error can't find the container with id 5ee6447607ee70e390d79d38ec16f7110cc30404389b2a5427bd077dced34c85 Jul 07 19:34:36.607688 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:36.607648 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" event={"ID":"e2317815-5702-4343-9ecf-91d3684b5ffa","Type":"ContainerStarted","Data":"5ee6447607ee70e390d79d38ec16f7110cc30404389b2a5427bd077dced34c85"} Jul 07 19:34:36.980776 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:36.980721 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:37.614505 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:37.614465 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" event={"ID":"e2317815-5702-4343-9ecf-91d3684b5ffa","Type":"ContainerStarted","Data":"a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7"} Jul 07 19:34:37.614999 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:37.614639 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:37.634383 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:37.634338 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" podStartSLOduration=2.3230443210000002 podStartE2EDuration="2.6343248s" totalImagesPullingTime="311.280479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:37.632086576 +0000 UTC m=+1583.567711941" watchObservedRunningTime="2026-07-07 19:34:37.6343248 +0000 UTC m=+1583.569950160" Jul 07 19:34:39.638075 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:39.638036 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:39.800753 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:39.800714 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6"] Jul 07 19:34:39.800940 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:39.800921 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" podUID="9924c5a8-0642-428c-ac9d-dd17b0acb5c4" containerName="istio-proxy" containerID="cri-o://9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17" gracePeriod=30 Jul 07 19:34:39.843675 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:39.843642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:39.965967 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:39.965856 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:40.080232 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.080195 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9"] Jul 07 19:34:40.080453 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.080427 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" podUID="e2317815-5702-4343-9ecf-91d3684b5ffa" containerName="maas-api" containerID="cri-o://a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7" gracePeriod=30 Jul 07 19:34:40.115460 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.115437 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:40.334375 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.334352 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:40.448660 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.448633 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2317815-5702-4343-9ecf-91d3684b5ffa-maas-api-tls\") pod \"e2317815-5702-4343-9ecf-91d3684b5ffa\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " Jul 07 19:34:40.448831 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.448707 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2317815-5702-4343-9ecf-91d3684b5ffa-openshift-service-ca\") pod \"e2317815-5702-4343-9ecf-91d3684b5ffa\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " Jul 07 19:34:40.448831 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.448752 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dflc\" (UniqueName: \"kubernetes.io/projected/e2317815-5702-4343-9ecf-91d3684b5ffa-kube-api-access-9dflc\") pod \"e2317815-5702-4343-9ecf-91d3684b5ffa\" (UID: \"e2317815-5702-4343-9ecf-91d3684b5ffa\") " Jul 07 19:34:40.449062 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.449038 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2317815-5702-4343-9ecf-91d3684b5ffa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2317815-5702-4343-9ecf-91d3684b5ffa" (UID: "e2317815-5702-4343-9ecf-91d3684b5ffa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:34:40.449216 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.449202 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2317815-5702-4343-9ecf-91d3684b5ffa-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.450838 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.450815 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2317815-5702-4343-9ecf-91d3684b5ffa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2317815-5702-4343-9ecf-91d3684b5ffa" (UID: "e2317815-5702-4343-9ecf-91d3684b5ffa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:34:40.450957 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.450855 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2317815-5702-4343-9ecf-91d3684b5ffa-kube-api-access-9dflc" (OuterVolumeSpecName: "kube-api-access-9dflc") pod "e2317815-5702-4343-9ecf-91d3684b5ffa" (UID: "e2317815-5702-4343-9ecf-91d3684b5ffa"). InnerVolumeSpecName "kube-api-access-9dflc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:40.550604 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.550509 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2317815-5702-4343-9ecf-91d3684b5ffa-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.550604 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.550546 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dflc\" (UniqueName: \"kubernetes.io/projected/e2317815-5702-4343-9ecf-91d3684b5ffa-kube-api-access-9dflc\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.630678 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.630643 2577 generic.go:356] "Generic (PLEG): container finished" podID="e2317815-5702-4343-9ecf-91d3684b5ffa" containerID="a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7" exitCode=0 Jul 07 19:34:40.630804 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.630690 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" Jul 07 19:34:40.630804 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.630698 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" event={"ID":"e2317815-5702-4343-9ecf-91d3684b5ffa","Type":"ContainerDied","Data":"a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7"} Jul 07 19:34:40.630804 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.630743 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9" event={"ID":"e2317815-5702-4343-9ecf-91d3684b5ffa","Type":"ContainerDied","Data":"5ee6447607ee70e390d79d38ec16f7110cc30404389b2a5427bd077dced34c85"} Jul 07 19:34:40.630804 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.630767 2577 scope.go:117] "RemoveContainer" containerID="a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7" Jul 07 19:34:40.639426 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.639407 2577 scope.go:117] "RemoveContainer" containerID="a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7" Jul 07 19:34:40.639701 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:40.639674 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7\": container with ID starting with a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7 not found: ID does not exist" containerID="a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7" Jul 07 19:34:40.639805 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.639707 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7"} err="failed to get container status \"a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7\": rpc error: code = NotFound desc = could not find container \"a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7\": container with ID starting with a712afebe4523fd145fe31dc7b810b4d3890a3cfdc9a27471ccff134d836efb7 not found: ID does not exist" Jul 07 19:34:40.648792 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.648769 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9"] Jul 07 19:34:40.652934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:40.652913 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-edcb1097-77cc8b7b98-mbgp9"] Jul 07 19:34:41.825380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:41.825346 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:42.604982 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:42.604949 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2317815-5702-4343-9ecf-91d3684b5ffa" path="/var/lib/kubelet/pods/e2317815-5702-4343-9ecf-91d3684b5ffa/volumes" Jul 07 19:34:45.068341 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.068287 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:45.195188 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195162 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istiod-ca-cert\") pod \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " Jul 07 19:34:45.195358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195212 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-envoy\") pod \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " Jul 07 19:34:45.195358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195239 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm796\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-kube-api-access-hm796\") pod \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " Jul 07 19:34:45.195358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195267 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-data\") pod \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " Jul 07 19:34:45.195358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195329 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-socket\") pod \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " Jul 07 19:34:45.195358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-credential-socket\") pod \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " Jul 07 19:34:45.195624 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195382 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-token\") pod \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " Jul 07 19:34:45.195624 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195432 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-podinfo\") pod \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " Jul 07 19:34:45.195624 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195481 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-certs\") pod \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\" (UID: \"9924c5a8-0642-428c-ac9d-dd17b0acb5c4\") " Jul 07 19:34:45.195779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195616 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "9924c5a8-0642-428c-ac9d-dd17b0acb5c4" (UID: "9924c5a8-0642-428c-ac9d-dd17b0acb5c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:45.195779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195634 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9924c5a8-0642-428c-ac9d-dd17b0acb5c4" (UID: "9924c5a8-0642-428c-ac9d-dd17b0acb5c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:45.195779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195623 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9924c5a8-0642-428c-ac9d-dd17b0acb5c4" (UID: "9924c5a8-0642-428c-ac9d-dd17b0acb5c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:34:45.195972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195821 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9924c5a8-0642-428c-ac9d-dd17b0acb5c4" (UID: "9924c5a8-0642-428c-ac9d-dd17b0acb5c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:45.195972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195861 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9924c5a8-0642-428c-ac9d-dd17b0acb5c4" (UID: "9924c5a8-0642-428c-ac9d-dd17b0acb5c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:45.195972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195943 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istiod-ca-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:45.195972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195966 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-data\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:45.196137 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.195979 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:45.197641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.197615 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-kube-api-access-hm796" (OuterVolumeSpecName: "kube-api-access-hm796") pod "9924c5a8-0642-428c-ac9d-dd17b0acb5c4" (UID: "9924c5a8-0642-428c-ac9d-dd17b0acb5c4"). InnerVolumeSpecName "kube-api-access-hm796". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:45.197775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.197640 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9924c5a8-0642-428c-ac9d-dd17b0acb5c4" (UID: "9924c5a8-0642-428c-ac9d-dd17b0acb5c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:45.197775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.197697 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "9924c5a8-0642-428c-ac9d-dd17b0acb5c4" (UID: "9924c5a8-0642-428c-ac9d-dd17b0acb5c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:45.198106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.198078 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9924c5a8-0642-428c-ac9d-dd17b0acb5c4" (UID: "9924c5a8-0642-428c-ac9d-dd17b0acb5c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:34:45.296852 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.296825 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-workload-certs\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:45.296852 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.296848 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-envoy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:45.296852 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.296856 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm796\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-kube-api-access-hm796\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:45.297066 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.296865 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-credential-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:45.297066 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.296900 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-token\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:45.297066 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.296908 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9924c5a8-0642-428c-ac9d-dd17b0acb5c4-istio-podinfo\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:34:45.658932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.658862 2577 generic.go:356] "Generic (PLEG): container finished" podID="9924c5a8-0642-428c-ac9d-dd17b0acb5c4" containerID="9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17" exitCode=0 Jul 07 19:34:45.659101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.658936 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" Jul 07 19:34:45.659101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.658954 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" event={"ID":"9924c5a8-0642-428c-ac9d-dd17b0acb5c4","Type":"ContainerDied","Data":"9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17"} Jul 07 19:34:45.659101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.659011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6" event={"ID":"9924c5a8-0642-428c-ac9d-dd17b0acb5c4","Type":"ContainerDied","Data":"5e4665eb1142206d0424030006195b49fe0acb4b7f46ce8b98a8d8df97b8a667"} Jul 07 19:34:45.659101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.659032 2577 scope.go:117] "RemoveContainer" containerID="9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17" Jul 07 19:34:45.668443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.668425 2577 scope.go:117] "RemoveContainer" containerID="9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17" Jul 07 19:34:45.668658 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:34:45.668642 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17\": container with ID starting with 9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17 not found: ID does not exist" containerID="9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17" Jul 07 19:34:45.668703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.668665 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17"} err="failed to get container status \"9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17\": rpc error: code = NotFound desc = could not find container \"9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17\": container with ID starting with 9e84c6c6d48f292637abeecdb488e6e3cb9fff82d76102a5e552619d9853bd17 not found: ID does not exist" Jul 07 19:34:45.680723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.680699 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6"] Jul 07 19:34:45.683253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.683234 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-edcb1097-openshift-default-5d55776787-npvx6"] Jul 07 19:34:45.843362 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.843326 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:45.945415 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:45.945340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:46.353918 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:46.353860 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:46.449404 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:46.449370 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:46.603840 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:46.603805 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9924c5a8-0642-428c-ac9d-dd17b0acb5c4" path="/var/lib/kubelet/pods/9924c5a8-0642-428c-ac9d-dd17b0acb5c4/volumes" Jul 07 19:34:47.877458 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:47.877428 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:53.838723 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:53.838687 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:53.943539 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:53.943505 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:54.039153 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:54.039115 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:54.143086 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:54.143005 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:54.507299 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:34:54.507264 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:20.435153 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:35:20.435116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:26.680151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:35:26.680113 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:26.781486 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:35:26.781449 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:26.880034 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:35:26.880000 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:42.817985 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:35:42.817949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:49.114586 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:35:49.114553 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:49.217392 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:35:49.217346 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:49.314809 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:35:49.314775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:59.802869 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:35:59.802819 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:00.576967 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:00.576926 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:04.248220 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:04.248185 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:06.633967 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:06.633928 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:06.704028 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:06.703998 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:06.800881 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:06.800850 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:06.900191 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:06.900108 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:07.001517 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:07.001478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:07.650955 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:07.650916 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:07.751037 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:07.751001 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:07.852981 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:07.852949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:09.517615 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:09.517573 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:15.417708 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:15.417672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:15.485711 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:15.485674 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:16.064750 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:16.064710 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:22.150314 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.150280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:22.249790 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.249755 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:22.372224 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.372188 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:22.654993 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.654966 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx"] Jul 07 19:36:22.655505 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.655488 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9924c5a8-0642-428c-ac9d-dd17b0acb5c4" containerName="istio-proxy" Jul 07 19:36:22.655590 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.655509 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9924c5a8-0642-428c-ac9d-dd17b0acb5c4" containerName="istio-proxy" Jul 07 19:36:22.655590 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.655517 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2317815-5702-4343-9ecf-91d3684b5ffa" containerName="maas-api" Jul 07 19:36:22.655590 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.655522 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2317815-5702-4343-9ecf-91d3684b5ffa" containerName="maas-api" Jul 07 19:36:22.655688 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.655593 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2317815-5702-4343-9ecf-91d3684b5ffa" containerName="maas-api" Jul 07 19:36:22.655688 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.655601 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9924c5a8-0642-428c-ac9d-dd17b0acb5c4" containerName="istio-proxy" Jul 07 19:36:22.658638 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.658623 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.661030 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.661010 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-39c6c169-openshift-default-dockercfg-7sf44\"" Jul 07 19:36:22.678608 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.678555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:22.716540 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.716511 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx"] Jul 07 19:36:22.725577 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.725548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-data\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.725742 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.725585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-token\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.725742 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.725678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-podinfo\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.725742 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.725718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pcv\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-kube-api-access-j4pcv\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.725951 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.725780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-envoy\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.725951 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.725807 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-certs\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.725951 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.725862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-credential-socket\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.725951 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.725925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f1a1a8-c5f3-4908-82e8-3b3741880536-istiod-ca-cert\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.726106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.725956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-socket\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.826712 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.826668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-data\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.826712 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.826718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-token\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.826994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.826757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-podinfo\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.826994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.826775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pcv\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-kube-api-access-j4pcv\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.826994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.826799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-envoy\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.826994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.826824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-certs\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.826994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.826862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-credential-socket\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.826994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.826932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f1a1a8-c5f3-4908-82e8-3b3741880536-istiod-ca-cert\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.826994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.826960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-socket\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.827353 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.827155 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-data\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.827353 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.827294 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-certs\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.827353 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.827329 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-credential-socket\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.827462 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.827353 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-socket\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.827998 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.827976 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f1a1a8-c5f3-4908-82e8-3b3741880536-istiod-ca-cert\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.829311 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.829280 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-envoy\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.829413 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.829374 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-podinfo\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.835430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.835411 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pcv\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-kube-api-access-j4pcv\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.835495 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.835452 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-token\") pod \"e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:22.969539 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:22.969453 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:23.136185 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:23.136158 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx"] Jul 07 19:36:23.137825 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:36:23.137789 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f1a1a8_c5f3_4908_82e8_3b3741880536.slice/crio-b95250a68b77ab84daac5e10ad647adcce681e5da835f2bbfb0af665b2ff28d9 WatchSource:0}: Error finding container b95250a68b77ab84daac5e10ad647adcce681e5da835f2bbfb0af665b2ff28d9: Status 404 returned error can't find the container with id b95250a68b77ab84daac5e10ad647adcce681e5da835f2bbfb0af665b2ff28d9 Jul 07 19:36:23.139908 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:23.139851 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:36:23.140020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:23.139942 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:36:23.140020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:23.139977 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:36:23.170402 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:23.170373 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" event={"ID":"55f1a1a8-c5f3-4908-82e8-3b3741880536","Type":"ContainerStarted","Data":"b95250a68b77ab84daac5e10ad647adcce681e5da835f2bbfb0af665b2ff28d9"} Jul 07 19:36:23.243289 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:23.243258 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:24.164342 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:24.164305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:24.177122 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:24.177091 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" event={"ID":"55f1a1a8-c5f3-4908-82e8-3b3741880536","Type":"ContainerStarted","Data":"18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6"} Jul 07 19:36:24.204800 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:24.204732 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" podStartSLOduration=2.204711399 podStartE2EDuration="2.204711399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:36:24.202791351 +0000 UTC m=+1690.138416713" watchObservedRunningTime="2026-07-07 19:36:24.204711399 +0000 UTC m=+1690.140336763" Jul 07 19:36:24.270483 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:24.270443 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:24.969609 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:24.969581 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:24.975254 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:24.975226 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:25.183863 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:25.183823 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:25.184931 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:25.184914 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:26.138474 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.138443 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:26.231047 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.231016 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb"] Jul 07 19:36:26.235066 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.235043 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.239049 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.239027 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-39c6c169-serving-cert\"" Jul 07 19:36:26.247446 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.247422 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb"] Jul 07 19:36:26.260110 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.260059 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:26.363537 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.363508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6g9\" (UniqueName: \"kubernetes.io/projected/ee9ba019-514f-48ff-bc01-0467598243e9-kube-api-access-cl6g9\") pod \"maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.363722 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.363569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee9ba019-514f-48ff-bc01-0467598243e9-maas-api-tls\") pod \"maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.363722 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.363597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9ba019-514f-48ff-bc01-0467598243e9-openshift-service-ca\") pod \"maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.371511 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.371476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:26.464240 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.464148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6g9\" (UniqueName: \"kubernetes.io/projected/ee9ba019-514f-48ff-bc01-0467598243e9-kube-api-access-cl6g9\") pod \"maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.464240 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.464208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee9ba019-514f-48ff-bc01-0467598243e9-maas-api-tls\") pod \"maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.464240 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.464228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9ba019-514f-48ff-bc01-0467598243e9-openshift-service-ca\") pod \"maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.464862 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.464845 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9ba019-514f-48ff-bc01-0467598243e9-openshift-service-ca\") pod \"maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.466778 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.466758 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee9ba019-514f-48ff-bc01-0467598243e9-maas-api-tls\") pod \"maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.494194 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.494162 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6g9\" (UniqueName: \"kubernetes.io/projected/ee9ba019-514f-48ff-bc01-0467598243e9-kube-api-access-cl6g9\") pod \"maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.545131 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.545090 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:26.547969 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.547939 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:26.700358 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:26.700326 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb"] Jul 07 19:36:26.703537 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:36:26.703515 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9ba019_514f_48ff_bc01_0467598243e9.slice/crio-828bd7f709b8b78f1342607450d775b148c4694d30c1cb8f883909bdde2c7234 WatchSource:0}: Error finding container 828bd7f709b8b78f1342607450d775b148c4694d30c1cb8f883909bdde2c7234: Status 404 returned error can't find the container with id 828bd7f709b8b78f1342607450d775b148c4694d30c1cb8f883909bdde2c7234 Jul 07 19:36:27.043942 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:27.043910 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:27.179202 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:27.179114 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:27.194464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:27.194432 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" event={"ID":"ee9ba019-514f-48ff-bc01-0467598243e9","Type":"ContainerStarted","Data":"828bd7f709b8b78f1342607450d775b148c4694d30c1cb8f883909bdde2c7234"} Jul 07 19:36:28.200792 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:28.200750 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" event={"ID":"ee9ba019-514f-48ff-bc01-0467598243e9","Type":"ContainerStarted","Data":"4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a"} Jul 07 19:36:28.201350 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:28.200917 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:28.220714 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:28.220672 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" podStartSLOduration=1.515593548 podStartE2EDuration="2.220657907s" totalImagesPullingTime="705.064359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:36:28.21863463 +0000 UTC m=+1694.154259991" watchObservedRunningTime="2026-07-07 19:36:28.220657907 +0000 UTC m=+1694.156283306" Jul 07 19:36:30.593529 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:30.593492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:31.253554 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:31.253518 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:31.422820 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:31.422780 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:31.714058 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:31.713961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:34.211844 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:34.211816 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:37.880719 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:37.880677 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:37.980636 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:37.980582 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:38.293921 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.293857 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:38.642411 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.642326 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb"] Jul 07 19:36:38.642596 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.642566 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" podUID="ee9ba019-514f-48ff-bc01-0467598243e9" containerName="maas-api" containerID="cri-o://4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a" gracePeriod=30 Jul 07 19:36:38.665721 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.665690 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:38.762545 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.762506 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:38.898351 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.898296 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:38.993684 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.993652 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee9ba019-514f-48ff-bc01-0467598243e9-maas-api-tls\") pod \"ee9ba019-514f-48ff-bc01-0467598243e9\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " Jul 07 19:36:38.994818 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.993748 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl6g9\" (UniqueName: \"kubernetes.io/projected/ee9ba019-514f-48ff-bc01-0467598243e9-kube-api-access-cl6g9\") pod \"ee9ba019-514f-48ff-bc01-0467598243e9\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " Jul 07 19:36:38.994818 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.993800 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9ba019-514f-48ff-bc01-0467598243e9-openshift-service-ca\") pod \"ee9ba019-514f-48ff-bc01-0467598243e9\" (UID: \"ee9ba019-514f-48ff-bc01-0467598243e9\") " Jul 07 19:36:38.994818 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.994133 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9ba019-514f-48ff-bc01-0467598243e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee9ba019-514f-48ff-bc01-0467598243e9" (UID: "ee9ba019-514f-48ff-bc01-0467598243e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:36:38.995742 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.995721 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9ba019-514f-48ff-bc01-0467598243e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee9ba019-514f-48ff-bc01-0467598243e9" (UID: "ee9ba019-514f-48ff-bc01-0467598243e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:36:38.995846 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:38.995821 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9ba019-514f-48ff-bc01-0467598243e9-kube-api-access-cl6g9" (OuterVolumeSpecName: "kube-api-access-cl6g9") pod "ee9ba019-514f-48ff-bc01-0467598243e9" (UID: "ee9ba019-514f-48ff-bc01-0467598243e9"). InnerVolumeSpecName "kube-api-access-cl6g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:36:39.094750 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.094721 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl6g9\" (UniqueName: \"kubernetes.io/projected/ee9ba019-514f-48ff-bc01-0467598243e9-kube-api-access-cl6g9\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:39.094750 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.094747 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee9ba019-514f-48ff-bc01-0467598243e9-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:39.094933 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.094757 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee9ba019-514f-48ff-bc01-0467598243e9-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:39.261404 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.261375 2577 generic.go:356] "Generic (PLEG): container finished" podID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerID="18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6" exitCode=0 Jul 07 19:36:39.261566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.261447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" event={"ID":"55f1a1a8-c5f3-4908-82e8-3b3741880536","Type":"ContainerDied","Data":"18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6"} Jul 07 19:36:39.261972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.261945 2577 scope.go:117] "RemoveContainer" containerID="18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6" Jul 07 19:36:39.262844 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.262823 2577 generic.go:356] "Generic (PLEG): container finished" podID="ee9ba019-514f-48ff-bc01-0467598243e9" containerID="4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a" exitCode=0 Jul 07 19:36:39.262967 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.262857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" event={"ID":"ee9ba019-514f-48ff-bc01-0467598243e9","Type":"ContainerDied","Data":"4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a"} Jul 07 19:36:39.262967 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.262903 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" Jul 07 19:36:39.262967 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.262933 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb" event={"ID":"ee9ba019-514f-48ff-bc01-0467598243e9","Type":"ContainerDied","Data":"828bd7f709b8b78f1342607450d775b148c4694d30c1cb8f883909bdde2c7234"} Jul 07 19:36:39.262967 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.262956 2577 scope.go:117] "RemoveContainer" containerID="4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a" Jul 07 19:36:39.263145 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.263000 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:36:39.263145 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.263082 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:36:39.263145 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.263129 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:36:39.278393 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.278375 2577 scope.go:117] "RemoveContainer" containerID="4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a" Jul 07 19:36:39.278677 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:36:39.278655 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a\": container with ID starting with 4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a not found: ID does not exist" containerID="4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a" Jul 07 19:36:39.278749 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.278689 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a"} err="failed to get container status \"4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a\": rpc error: code = NotFound desc = could not find container \"4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a\": container with ID starting with 4b171f2a62174ffa527a474e490109c33965d77467d32f47e6fd045b15e8cf1a not found: ID does not exist" Jul 07 19:36:39.301392 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.301369 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb"] Jul 07 19:36:39.303828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:39.303803 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-39c6c169-5b9895bc8c-5trcb"] Jul 07 19:36:40.184437 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:40.184400 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:40.270350 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:40.270313 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" event={"ID":"55f1a1a8-c5f3-4908-82e8-3b3741880536","Type":"ContainerStarted","Data":"251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba"} Jul 07 19:36:40.608076 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:40.608041 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee9ba019-514f-48ff-bc01-0467598243e9" path="/var/lib/kubelet/pods/ee9ba019-514f-48ff-bc01-0467598243e9/volumes" Jul 07 19:36:40.970502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:40.970422 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:40.971500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:40.971478 2577 patch_prober.go:28] interesting pod/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 19:36:40.971558 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:40.971527 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 19:36:41.970735 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:41.970701 2577 patch_prober.go:28] interesting pod/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 19:36:41.971142 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:41.970752 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 19:36:42.970543 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:42.970506 2577 patch_prober.go:28] interesting pod/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 19:36:42.970754 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:42.970569 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 19:36:43.970370 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:43.970336 2577 patch_prober.go:28] interesting pod/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 07 19:36:43.970531 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:43.970391 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 07 19:36:44.839649 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:44.839612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:44.939366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:44.939329 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:44.973923 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:44.973896 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:44.974262 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:44.974244 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:44.974904 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:44.974868 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:45.070953 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:45.070923 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx"] Jul 07 19:36:45.087040 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:45.087003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:45.303003 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:45.302968 2577 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" secret="" err="secret \"e2e-mt-39c6c169-openshift-default-dockercfg-7sf44\" not found" Jul 07 19:36:45.307244 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:45.307216 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:45.433304 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:45.433260 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:45.523363 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:45.523328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:45.628147 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:45.628065 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:46.308414 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:46.308372 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" containerID="cri-o://251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba" gracePeriod=30 Jul 07 19:36:51.560031 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.559996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:51.570162 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.570143 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:51.618711 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.618632 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:51.627762 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.627741 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-certs\") pod \"55f1a1a8-c5f3-4908-82e8-3b3741880536\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " Jul 07 19:36:51.627888 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.627792 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-data\") pod \"55f1a1a8-c5f3-4908-82e8-3b3741880536\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " Jul 07 19:36:51.627888 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.627819 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-credential-socket\") pod \"55f1a1a8-c5f3-4908-82e8-3b3741880536\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " Jul 07 19:36:51.627888 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.627844 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f1a1a8-c5f3-4908-82e8-3b3741880536-istiod-ca-cert\") pod \"55f1a1a8-c5f3-4908-82e8-3b3741880536\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " Jul 07 19:36:51.628012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.627903 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4pcv\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-kube-api-access-j4pcv\") pod \"55f1a1a8-c5f3-4908-82e8-3b3741880536\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " Jul 07 19:36:51.628012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.627934 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-podinfo\") pod \"55f1a1a8-c5f3-4908-82e8-3b3741880536\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " Jul 07 19:36:51.628012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.627973 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-token\") pod \"55f1a1a8-c5f3-4908-82e8-3b3741880536\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " Jul 07 19:36:51.628164 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628008 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "55f1a1a8-c5f3-4908-82e8-3b3741880536" (UID: "55f1a1a8-c5f3-4908-82e8-3b3741880536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:51.628164 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628083 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-socket\") pod \"55f1a1a8-c5f3-4908-82e8-3b3741880536\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " Jul 07 19:36:51.628164 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628107 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "55f1a1a8-c5f3-4908-82e8-3b3741880536" (UID: "55f1a1a8-c5f3-4908-82e8-3b3741880536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:51.628164 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628122 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-envoy\") pod \"55f1a1a8-c5f3-4908-82e8-3b3741880536\" (UID: \"55f1a1a8-c5f3-4908-82e8-3b3741880536\") " Jul 07 19:36:51.628374 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628279 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f1a1a8-c5f3-4908-82e8-3b3741880536-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "55f1a1a8-c5f3-4908-82e8-3b3741880536" (UID: "55f1a1a8-c5f3-4908-82e8-3b3741880536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:36:51.628437 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628347 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "55f1a1a8-c5f3-4908-82e8-3b3741880536" (UID: "55f1a1a8-c5f3-4908-82e8-3b3741880536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:51.628522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628500 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-certs\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:51.628583 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628526 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-credential-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:51.628583 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628542 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f1a1a8-c5f3-4908-82e8-3b3741880536-istiod-ca-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:51.628583 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628557 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-workload-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:51.628752 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.628731 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-data" (OuterVolumeSpecName: "istio-data") pod "55f1a1a8-c5f3-4908-82e8-3b3741880536" (UID: "55f1a1a8-c5f3-4908-82e8-3b3741880536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:51.630198 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.630172 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-kube-api-access-j4pcv" (OuterVolumeSpecName: "kube-api-access-j4pcv") pod "55f1a1a8-c5f3-4908-82e8-3b3741880536" (UID: "55f1a1a8-c5f3-4908-82e8-3b3741880536"). InnerVolumeSpecName "kube-api-access-j4pcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:36:51.630439 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.630415 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "55f1a1a8-c5f3-4908-82e8-3b3741880536" (UID: "55f1a1a8-c5f3-4908-82e8-3b3741880536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:51.630744 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.630728 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-token" (OuterVolumeSpecName: "istio-token") pod "55f1a1a8-c5f3-4908-82e8-3b3741880536" (UID: "55f1a1a8-c5f3-4908-82e8-3b3741880536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:36:51.630804 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.630755 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "55f1a1a8-c5f3-4908-82e8-3b3741880536" (UID: "55f1a1a8-c5f3-4908-82e8-3b3741880536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:36:51.688682 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.688648 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:51.729915 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.729893 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-envoy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:51.729915 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.729913 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-data\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:51.730095 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.729923 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4pcv\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-kube-api-access-j4pcv\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:51.730095 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.729934 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-podinfo\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:51.730095 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.729943 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f1a1a8-c5f3-4908-82e8-3b3741880536-istio-token\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:36:51.788516 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.788477 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:51.920748 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:51.920660 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:52.341623 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.341589 2577 generic.go:356] "Generic (PLEG): container finished" podID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerID="251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba" exitCode=0 Jul 07 19:36:52.341815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.341632 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" Jul 07 19:36:52.341815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.341675 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" event={"ID":"55f1a1a8-c5f3-4908-82e8-3b3741880536","Type":"ContainerDied","Data":"251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba"} Jul 07 19:36:52.341815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.341726 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx" event={"ID":"55f1a1a8-c5f3-4908-82e8-3b3741880536","Type":"ContainerDied","Data":"b95250a68b77ab84daac5e10ad647adcce681e5da835f2bbfb0af665b2ff28d9"} Jul 07 19:36:52.341815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.341751 2577 scope.go:117] "RemoveContainer" containerID="251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba" Jul 07 19:36:52.351668 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.351653 2577 scope.go:117] "RemoveContainer" containerID="18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6" Jul 07 19:36:52.360490 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.360474 2577 scope.go:117] "RemoveContainer" containerID="251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba" Jul 07 19:36:52.360722 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:36:52.360704 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba\": container with ID starting with 251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba not found: ID does not exist" containerID="251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba" Jul 07 19:36:52.360765 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.360732 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba"} err="failed to get container status \"251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba\": rpc error: code = NotFound desc = could not find container \"251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba\": container with ID starting with 251393878f9a000569659a710811cdf817337045170837267599186f3a4de5ba not found: ID does not exist" Jul 07 19:36:52.360765 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.360748 2577 scope.go:117] "RemoveContainer" containerID="18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6" Jul 07 19:36:52.360982 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:36:52.360967 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6\": container with ID starting with 18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6 not found: ID does not exist" containerID="18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6" Jul 07 19:36:52.361044 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.360989 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6"} err="failed to get container status \"18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6\": rpc error: code = NotFound desc = could not find container \"18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6\": container with ID starting with 18acdb98f4ecd7ff6b849f7adfd7fe76cfb8aefe3f1ef652634582c9ff730da6 not found: ID does not exist" Jul 07 19:36:52.365543 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.365523 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx"] Jul 07 19:36:52.369971 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.369951 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-39c6c169-openshift-default-5c44f8b9ff-ll8gx"] Jul 07 19:36:52.604130 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:52.604052 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" path="/var/lib/kubelet/pods/55f1a1a8-c5f3-4908-82e8-3b3741880536/volumes" Jul 07 19:36:58.019449 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:58.019412 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:58.081724 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:36:58.081695 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:04.625807 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:04.625768 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:04.726217 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:04.726188 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:04.826087 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:04.826039 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:05.004048 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.004020 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f"] Jul 07 19:37:05.004450 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.004438 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" Jul 07 19:37:05.004497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.004452 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" Jul 07 19:37:05.004497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.004460 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" Jul 07 19:37:05.004497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.004465 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" Jul 07 19:37:05.004497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.004480 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee9ba019-514f-48ff-bc01-0467598243e9" containerName="maas-api" Jul 07 19:37:05.004497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.004485 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee9ba019-514f-48ff-bc01-0467598243e9" containerName="maas-api" Jul 07 19:37:05.004646 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.004548 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee9ba019-514f-48ff-bc01-0467598243e9" containerName="maas-api" Jul 07 19:37:05.004646 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.004555 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" Jul 07 19:37:05.008182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.008166 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.010790 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.010767 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-adb1679d-openshift-default-dockercfg-v54nf\"" Jul 07 19:37:05.022618 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.022600 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f"] Jul 07 19:37:05.034845 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.034827 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:05.058324 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.058294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-data\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.058400 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.058329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8nj\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-kube-api-access-cb8nj\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.058446 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.058423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005a3844-adce-4e19-8fb5-786f64b55121-istiod-ca-cert\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.058484 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.058463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-socket\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.058522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.058490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-credential-socket\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.058522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.058511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-envoy\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.058586 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.058544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005a3844-adce-4e19-8fb5-786f64b55121-istio-podinfo\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.058586 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.058569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-certs\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.058649 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.058626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-istio-token\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.130183 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.130154 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:05.159288 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-credential-socket\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159403 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-envoy\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159403 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005a3844-adce-4e19-8fb5-786f64b55121-istio-podinfo\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159403 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-certs\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-istio-token\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-data\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159669 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8nj\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-kube-api-access-cb8nj\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159728 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159666 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-credential-socket\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159728 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005a3844-adce-4e19-8fb5-786f64b55121-istiod-ca-cert\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159728 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-socket\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159771 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-certs\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.159972 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.159808 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-data\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.160078 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.160060 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-socket\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.160637 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.160615 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005a3844-adce-4e19-8fb5-786f64b55121-istiod-ca-cert\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.161820 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.161795 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-envoy\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.161937 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.161921 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005a3844-adce-4e19-8fb5-786f64b55121-istio-podinfo\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.167496 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.167476 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-istio-token\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.167596 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.167500 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8nj\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-kube-api-access-cb8nj\") pod \"e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.318909 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.318798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:05.462477 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.462449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f"] Jul 07 19:37:05.464518 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:37:05.464488 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005a3844_adce_4e19_8fb5_786f64b55121.slice/crio-b1524e83133daab021426ef978cd491fbf6bdfb0d471ef109586172d0354b853 WatchSource:0}: Error finding container b1524e83133daab021426ef978cd491fbf6bdfb0d471ef109586172d0354b853: Status 404 returned error can't find the container with id b1524e83133daab021426ef978cd491fbf6bdfb0d471ef109586172d0354b853 Jul 07 19:37:05.466661 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.466623 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:37:05.466712 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.466688 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:37:05.466747 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:05.466714 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:37:06.415845 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:06.415805 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" event={"ID":"005a3844-adce-4e19-8fb5-786f64b55121","Type":"ContainerStarted","Data":"2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446"} Jul 07 19:37:06.416248 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:06.415896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" event={"ID":"005a3844-adce-4e19-8fb5-786f64b55121","Type":"ContainerStarted","Data":"b1524e83133daab021426ef978cd491fbf6bdfb0d471ef109586172d0354b853"} Jul 07 19:37:06.436101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:06.436050 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" podStartSLOduration=2.436033433 podStartE2EDuration="2.436033433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:37:06.435112791 +0000 UTC m=+1732.370738151" watchObservedRunningTime="2026-07-07 19:37:06.436033433 +0000 UTC m=+1732.371658794" Jul 07 19:37:07.319105 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:07.319068 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:07.324328 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:07.324302 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:07.420825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:07.420795 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:07.421914 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:07.421867 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:08.306773 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.306737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:08.396950 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.396863 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv"] Jul 07 19:37:08.397533 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.397513 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f1a1a8-c5f3-4908-82e8-3b3741880536" containerName="istio-proxy" Jul 07 19:37:08.400176 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.400154 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:08.403070 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.403039 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-adb1679d-serving-cert\"" Jul 07 19:37:08.410578 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.410105 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv"] Jul 07 19:37:08.412034 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.411986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:08.494827 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.494783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bvj4\" (UniqueName: \"kubernetes.io/projected/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-kube-api-access-9bvj4\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:08.495348 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.494900 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-openshift-service-ca\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:08.495348 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.495161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-maas-api-tls\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:08.509533 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.509496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:08.596410 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.596321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bvj4\" (UniqueName: \"kubernetes.io/projected/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-kube-api-access-9bvj4\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:08.596410 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.596385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-openshift-service-ca\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:08.596639 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.596462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-maas-api-tls\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:08.596639 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:37:08.596541 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-adb1679d-serving-cert: secret "maas-api-e2e-mt-adb1679d-serving-cert" not found Jul 07 19:37:08.596639 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:37:08.596625 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-maas-api-tls podName:e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a nodeName:}" failed. No retries permitted until 2026-07-07 19:37:09.096609158 +0000 UTC m=+1735.032234499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-maas-api-tls") pod "maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" (UID: "e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a") : secret "maas-api-e2e-mt-adb1679d-serving-cert" not found Jul 07 19:37:08.597055 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.597035 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-openshift-service-ca\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:08.611566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.610591 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bvj4\" (UniqueName: \"kubernetes.io/projected/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-kube-api-access-9bvj4\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:08.728101 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:08.728073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:09.009310 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:09.009267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:09.100608 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:09.100580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-maas-api-tls\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:09.103133 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:09.103101 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-maas-api-tls\") pod \"maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:09.110996 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:09.110966 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:09.314273 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:09.314190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:09.457148 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:09.457121 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv"] Jul 07 19:37:09.461150 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:37:09.461121 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21ed6c3_9c8d_47f3_b9f5_455d31e78a7a.slice/crio-bfd729c30aa79197e398e0c4a01153f20b44863e50466f764f74653eb28a419c WatchSource:0}: Error finding container bfd729c30aa79197e398e0c4a01153f20b44863e50466f764f74653eb28a419c: Status 404 returned error can't find the container with id bfd729c30aa79197e398e0c4a01153f20b44863e50466f764f74653eb28a419c Jul 07 19:37:10.439472 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:10.439438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" event={"ID":"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a","Type":"ContainerStarted","Data":"cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b"} Jul 07 19:37:10.439966 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:10.439492 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" event={"ID":"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a","Type":"ContainerStarted","Data":"bfd729c30aa79197e398e0c4a01153f20b44863e50466f764f74653eb28a419c"} Jul 07 19:37:10.439966 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:10.439519 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:10.459553 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:10.459507 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" podStartSLOduration=2.068716067 podStartE2EDuration="2.459492089s" totalImagesPullingTime="390.776022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:37:10.458138383 +0000 UTC m=+1736.393763744" watchObservedRunningTime="2026-07-07 19:37:10.459492089 +0000 UTC m=+1736.395117448" Jul 07 19:37:12.790256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:12.790217 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:16.450148 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:16.450117 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:37:23.344978 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:23.344943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:23.462056 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:23.462021 2577 patch_prober.go:28] interesting pod/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 19:37:23.462235 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:23.462083 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 19:37:24.521558 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:24.521526 2577 generic.go:356] "Generic (PLEG): container finished" podID="005a3844-adce-4e19-8fb5-786f64b55121" containerID="2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446" exitCode=0 Jul 07 19:37:24.522096 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:24.521606 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" event={"ID":"005a3844-adce-4e19-8fb5-786f64b55121","Type":"ContainerDied","Data":"2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446"} Jul 07 19:37:24.522166 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:24.522117 2577 scope.go:117] "RemoveContainer" containerID="2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446" Jul 07 19:37:24.523144 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:24.523117 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:37:24.523213 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:24.523183 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:37:24.523258 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:24.523214 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:37:25.528693 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:25.528657 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" event={"ID":"005a3844-adce-4e19-8fb5-786f64b55121","Type":"ContainerStarted","Data":"4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91"} Jul 07 19:37:26.319441 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:26.319410 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:26.319652 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:26.319612 2577 patch_prober.go:28] interesting pod/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 07 19:37:26.319731 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:26.319660 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 07 19:37:27.319270 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:27.319234 2577 patch_prober.go:28] interesting pod/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 07 19:37:27.319657 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:27.319289 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 07 19:37:28.319595 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:28.319554 2577 patch_prober.go:28] interesting pod/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 07 19:37:28.320004 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:28.319619 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 07 19:37:29.319336 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:29.319298 2577 patch_prober.go:28] interesting pod/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 07 19:37:29.319501 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:29.319359 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 07 19:37:30.323056 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:30.323002 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:30.323433 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:30.323347 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:37:30.323969 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:37:30.323952 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:38:00.538130 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:00.538086 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:00.994260 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:00.994229 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv"] Jul 07 19:38:00.994543 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:00.994492 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" podUID="e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a" containerName="maas-api" containerID="cri-o://cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b" gracePeriod=30 Jul 07 19:38:01.033925 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.033852 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:01.248757 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.248701 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:38:01.429830 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.429784 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-maas-api-tls\") pod \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " Jul 07 19:38:01.430054 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.429857 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bvj4\" (UniqueName: \"kubernetes.io/projected/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-kube-api-access-9bvj4\") pod \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " Jul 07 19:38:01.430054 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.429924 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-openshift-service-ca\") pod \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\" (UID: \"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a\") " Jul 07 19:38:01.430285 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.430262 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a" (UID: "e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:38:01.432045 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.432023 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a" (UID: "e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:38:01.432150 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.432104 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-kube-api-access-9bvj4" (OuterVolumeSpecName: "kube-api-access-9bvj4") pod "e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a" (UID: "e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a"). InnerVolumeSpecName "kube-api-access-9bvj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:38:01.531368 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.531302 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:38:01.531368 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.531329 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bvj4\" (UniqueName: \"kubernetes.io/projected/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-kube-api-access-9bvj4\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:38:01.531368 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.531345 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:38:01.716495 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.716459 2577 generic.go:356] "Generic (PLEG): container finished" podID="e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a" containerID="cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b" exitCode=0 Jul 07 19:38:01.716495 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.716497 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" Jul 07 19:38:01.716993 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.716509 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" event={"ID":"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a","Type":"ContainerDied","Data":"cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b"} Jul 07 19:38:01.716993 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.716559 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv" event={"ID":"e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a","Type":"ContainerDied","Data":"bfd729c30aa79197e398e0c4a01153f20b44863e50466f764f74653eb28a419c"} Jul 07 19:38:01.716993 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.716580 2577 scope.go:117] "RemoveContainer" containerID="cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b" Jul 07 19:38:01.726189 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.726171 2577 scope.go:117] "RemoveContainer" containerID="cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b" Jul 07 19:38:01.726432 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:38:01.726417 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b\": container with ID starting with cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b not found: ID does not exist" containerID="cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b" Jul 07 19:38:01.726486 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.726440 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b"} err="failed to get container status \"cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b\": rpc error: code = NotFound desc = could not find container \"cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b\": container with ID starting with cbe6ed19130b98352539c36b2701cdf478ed6f96a6c2264b308858abd73b5a0b not found: ID does not exist" Jul 07 19:38:01.740702 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.740673 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv"] Jul 07 19:38:01.746002 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:01.745982 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-adb1679d-6ff7ccbb45-tgxgv"] Jul 07 19:38:02.604774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:02.604746 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a" path="/var/lib/kubelet/pods/e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a/volumes" Jul 07 19:38:06.582772 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:06.582679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:11.613631 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:11.613598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:48.289464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:48.289421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:48.987799 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:48.987761 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:49.990607 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:49.990574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:50.063693 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:50.063660 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:54.984402 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:54.984357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:55.047139 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:55.047100 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:55.243566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:55.243544 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f"] Jul 07 19:38:55.243767 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:55.243742 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" containerID="cri-o://4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91" gracePeriod=30 Jul 07 19:38:55.274372 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:55.274342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:55.485683 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:55.485648 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:56.538485 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:56.538449 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:56.640266 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:56.640232 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:56.737680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:56.737635 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:57.458680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:57.458642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:57.556687 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:57.556648 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:57.655186 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:57.655156 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:59.004096 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:38:59.004067 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:00.324672 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.324641 2577 patch_prober.go:28] interesting pod/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 07 19:39:00.325016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.324704 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 07 19:39:00.498229 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.498206 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:39:00.576773 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.576697 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-data\") pod \"005a3844-adce-4e19-8fb5-786f64b55121\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " Jul 07 19:39:00.576773 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.576729 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-certs\") pod \"005a3844-adce-4e19-8fb5-786f64b55121\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " Jul 07 19:39:00.577026 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.576775 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-socket\") pod \"005a3844-adce-4e19-8fb5-786f64b55121\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " Jul 07 19:39:00.577026 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.576807 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005a3844-adce-4e19-8fb5-786f64b55121-istiod-ca-cert\") pod \"005a3844-adce-4e19-8fb5-786f64b55121\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " Jul 07 19:39:00.577026 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.576843 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb8nj\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-kube-api-access-cb8nj\") pod \"005a3844-adce-4e19-8fb5-786f64b55121\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " Jul 07 19:39:00.577026 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.576919 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-credential-socket\") pod \"005a3844-adce-4e19-8fb5-786f64b55121\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " Jul 07 19:39:00.577249 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577043 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-envoy\") pod \"005a3844-adce-4e19-8fb5-786f64b55121\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " Jul 07 19:39:00.577249 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577076 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "005a3844-adce-4e19-8fb5-786f64b55121" (UID: "005a3844-adce-4e19-8fb5-786f64b55121"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:39:00.577249 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577093 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005a3844-adce-4e19-8fb5-786f64b55121-istio-podinfo\") pod \"005a3844-adce-4e19-8fb5-786f64b55121\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " Jul 07 19:39:00.577249 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577142 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-istio-token\") pod \"005a3844-adce-4e19-8fb5-786f64b55121\" (UID: \"005a3844-adce-4e19-8fb5-786f64b55121\") " Jul 07 19:39:00.577249 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577142 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "005a3844-adce-4e19-8fb5-786f64b55121" (UID: "005a3844-adce-4e19-8fb5-786f64b55121"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:39:00.577498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577255 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "005a3844-adce-4e19-8fb5-786f64b55121" (UID: "005a3844-adce-4e19-8fb5-786f64b55121"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:39:00.577498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577311 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005a3844-adce-4e19-8fb5-786f64b55121-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "005a3844-adce-4e19-8fb5-786f64b55121" (UID: "005a3844-adce-4e19-8fb5-786f64b55121"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:39:00.577595 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577531 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:39:00.577595 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577553 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005a3844-adce-4e19-8fb5-786f64b55121-istiod-ca-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:39:00.577595 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577572 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-credential-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:39:00.577595 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577586 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-workload-certs\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:39:00.577983 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.577707 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-data" (OuterVolumeSpecName: "istio-data") pod "005a3844-adce-4e19-8fb5-786f64b55121" (UID: "005a3844-adce-4e19-8fb5-786f64b55121"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:39:00.579363 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.579335 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-istio-token" (OuterVolumeSpecName: "istio-token") pod "005a3844-adce-4e19-8fb5-786f64b55121" (UID: "005a3844-adce-4e19-8fb5-786f64b55121"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:39:00.579769 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.579739 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/005a3844-adce-4e19-8fb5-786f64b55121-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "005a3844-adce-4e19-8fb5-786f64b55121" (UID: "005a3844-adce-4e19-8fb5-786f64b55121"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:39:00.579769 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.579764 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-kube-api-access-cb8nj" (OuterVolumeSpecName: "kube-api-access-cb8nj") pod "005a3844-adce-4e19-8fb5-786f64b55121" (UID: "005a3844-adce-4e19-8fb5-786f64b55121"). InnerVolumeSpecName "kube-api-access-cb8nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:39:00.579932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.579830 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "005a3844-adce-4e19-8fb5-786f64b55121" (UID: "005a3844-adce-4e19-8fb5-786f64b55121"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:39:00.678813 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.678781 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8nj\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-kube-api-access-cb8nj\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:39:00.678813 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.678809 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-envoy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:39:00.679019 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.678821 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005a3844-adce-4e19-8fb5-786f64b55121-istio-podinfo\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:39:00.679019 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.678833 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005a3844-adce-4e19-8fb5-786f64b55121-istio-token\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:39:00.679019 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:00.678844 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005a3844-adce-4e19-8fb5-786f64b55121-istio-data\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:39:01.030306 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.030277 2577 generic.go:356] "Generic (PLEG): container finished" podID="005a3844-adce-4e19-8fb5-786f64b55121" containerID="4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91" exitCode=0 Jul 07 19:39:01.030458 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.030327 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" Jul 07 19:39:01.030458 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.030357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" event={"ID":"005a3844-adce-4e19-8fb5-786f64b55121","Type":"ContainerDied","Data":"4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91"} Jul 07 19:39:01.030458 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.030406 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f" event={"ID":"005a3844-adce-4e19-8fb5-786f64b55121","Type":"ContainerDied","Data":"b1524e83133daab021426ef978cd491fbf6bdfb0d471ef109586172d0354b853"} Jul 07 19:39:01.030458 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.030429 2577 scope.go:117] "RemoveContainer" containerID="4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91" Jul 07 19:39:01.039591 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.039573 2577 scope.go:117] "RemoveContainer" containerID="2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446" Jul 07 19:39:01.047758 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.047712 2577 scope.go:117] "RemoveContainer" containerID="4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91" Jul 07 19:39:01.048010 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:39:01.047990 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91\": container with ID starting with 4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91 not found: ID does not exist" containerID="4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91" Jul 07 19:39:01.048094 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.048020 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91"} err="failed to get container status \"4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91\": rpc error: code = NotFound desc = could not find container \"4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91\": container with ID starting with 4d778014d3d2d48b6ec4ad7dc34ba1710934d1bc9b52cb426c523a194209bf91 not found: ID does not exist" Jul 07 19:39:01.048094 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.048046 2577 scope.go:117] "RemoveContainer" containerID="2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446" Jul 07 19:39:01.048293 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:39:01.048275 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446\": container with ID starting with 2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446 not found: ID does not exist" containerID="2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446" Jul 07 19:39:01.048336 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.048301 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446"} err="failed to get container status \"2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446\": rpc error: code = NotFound desc = could not find container \"2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446\": container with ID starting with 2d748bac0357a7f766e94fcdae4b41449da1a989eb8404bead1d881ed7c87446 not found: ID does not exist" Jul 07 19:39:01.055742 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.055695 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f"] Jul 07 19:39:01.057924 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:01.057905 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-adb1679d-openshift-default-69db98f76c-6hk5f"] Jul 07 19:39:02.604317 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:02.604285 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="005a3844-adce-4e19-8fb5-786f64b55121" path="/var/lib/kubelet/pods/005a3844-adce-4e19-8fb5-786f64b55121/volumes" Jul 07 19:39:05.198629 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:05.198591 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:05.299087 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:05.299057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:05.854322 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:05.854283 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:12.014823 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:12.014784 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:12.130582 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:12.130543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:12.217789 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:12.217748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:22.709024 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:22.708979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:23.057736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:23.057703 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:43.971749 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:43.971668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:50.104512 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.104480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:50.185197 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.185153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:50.408977 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.408863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:50.694411 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.694341 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc"] Jul 07 19:39:50.694773 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.694761 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" Jul 07 19:39:50.694815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.694775 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" Jul 07 19:39:50.694815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.694787 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" Jul 07 19:39:50.694815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.694792 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" Jul 07 19:39:50.694815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.694799 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a" containerName="maas-api" Jul 07 19:39:50.694815 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.694804 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a" containerName="maas-api" Jul 07 19:39:50.695001 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.694901 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" Jul 07 19:39:50.695001 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.694915 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21ed6c3-9c8d-47f3-b9f5-455d31e78a7a" containerName="maas-api" Jul 07 19:39:50.698083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.698066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.702038 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.702011 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-444f1d-openshift-default-dockercfg-f6rt7\"" Jul 07 19:39:50.714683 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.714662 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc"] Jul 07 19:39:50.720202 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.720177 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:50.755006 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.754973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-podinfo\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.755156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.755009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d8jf\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-kube-api-access-9d8jf\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.755156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.755079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istiod-ca-cert\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.755156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.755149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-token\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.755276 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.755183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-socket\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.755276 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.755212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-data\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.755276 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.755233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-envoy\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.755276 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.755252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-certs\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.755423 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.755285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-credential-socket\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.848552 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.848514 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:50.855744 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.855723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-data\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.855858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.855751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-envoy\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.855858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.855773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-certs\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.855858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.855799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-credential-socket\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856018 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.855860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-podinfo\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856096 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.856068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d8jf\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-kube-api-access-9d8jf\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856155 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.856132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istiod-ca-cert\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856213 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.856160 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-data\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.856223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-token\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.856251 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-certs\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.856272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-socket\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.856312 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-credential-socket\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856776 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.856756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-socket\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.856822 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.856768 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istiod-ca-cert\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.858328 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.858306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-envoy\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.858421 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.858405 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-podinfo\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.864919 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.864894 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d8jf\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-kube-api-access-9d8jf\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:50.865407 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:50.865391 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-token\") pod \"e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:51.008602 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:51.008569 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:51.150142 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:51.150120 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc"] Jul 07 19:39:51.152255 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:39:51.152225 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe231d27_7b7d_4f81_ab6e_b32f2905b028.slice/crio-3ebadae10cdee16d63518c9da5ae61a611ba06ccec731787e9b44622db260fee WatchSource:0}: Error finding container 3ebadae10cdee16d63518c9da5ae61a611ba06ccec731787e9b44622db260fee: Status 404 returned error can't find the container with id 3ebadae10cdee16d63518c9da5ae61a611ba06ccec731787e9b44622db260fee Jul 07 19:39:51.154473 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:51.154436 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:39:51.154563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:51.154500 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:39:51.154563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:51.154529 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:39:51.296868 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:51.296770 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" event={"ID":"fe231d27-7b7d-4f81-ab6e-b32f2905b028","Type":"ContainerStarted","Data":"54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d"} Jul 07 19:39:51.296868 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:51.296833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" event={"ID":"fe231d27-7b7d-4f81-ab6e-b32f2905b028","Type":"ContainerStarted","Data":"3ebadae10cdee16d63518c9da5ae61a611ba06ccec731787e9b44622db260fee"} Jul 07 19:39:51.318806 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:51.318733 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" podStartSLOduration=1.318711626 podStartE2EDuration="1.318711626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:39:51.317427072 +0000 UTC m=+1897.253052432" watchObservedRunningTime="2026-07-07 19:39:51.318711626 +0000 UTC m=+1897.254336988" Jul 07 19:39:52.009657 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:52.009621 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:52.014587 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:52.014564 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:52.302598 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:52.302519 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:52.303444 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:52.303426 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:39:53.246145 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.246110 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:53.316710 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.316672 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj"] Jul 07 19:39:53.317551 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.317531 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="005a3844-adce-4e19-8fb5-786f64b55121" containerName="istio-proxy" Jul 07 19:39:53.321149 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.321130 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.323747 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.323720 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-444f1d-serving-cert\"" Jul 07 19:39:53.332651 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.332628 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj"] Jul 07 19:39:53.333950 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.333924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:53.382658 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.382627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e36ef8-a7d3-4189-a987-8e709e49c9e3-maas-api-tls\") pod \"maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.382658 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.382659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e36ef8-a7d3-4189-a987-8e709e49c9e3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.382909 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.382692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rb9d\" (UniqueName: \"kubernetes.io/projected/83e36ef8-a7d3-4189-a987-8e709e49c9e3-kube-api-access-9rb9d\") pod \"maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.410597 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.410574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:53.484393 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.484356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e36ef8-a7d3-4189-a987-8e709e49c9e3-maas-api-tls\") pod \"maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.484575 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.484403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e36ef8-a7d3-4189-a987-8e709e49c9e3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.484575 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.484441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rb9d\" (UniqueName: \"kubernetes.io/projected/83e36ef8-a7d3-4189-a987-8e709e49c9e3-kube-api-access-9rb9d\") pod \"maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.485168 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.485145 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e36ef8-a7d3-4189-a987-8e709e49c9e3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.486828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.486809 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e36ef8-a7d3-4189-a987-8e709e49c9e3-maas-api-tls\") pod \"maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.493543 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.493520 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rb9d\" (UniqueName: \"kubernetes.io/projected/83e36ef8-a7d3-4189-a987-8e709e49c9e3-kube-api-access-9rb9d\") pod \"maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.510085 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.510020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:53.633826 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.633795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:53.816439 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.816354 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:53.916070 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.916037 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:53.988774 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.988733 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj"] Jul 07 19:39:53.990754 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:39:53.990728 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e36ef8_a7d3_4189_a987_8e709e49c9e3.slice/crio-d933a573ee2e2ee48d8f615b045b92ca5585a6134453ae1affb7d85fb2046386 WatchSource:0}: Error finding container d933a573ee2e2ee48d8f615b045b92ca5585a6134453ae1affb7d85fb2046386: Status 404 returned error can't find the container with id d933a573ee2e2ee48d8f615b045b92ca5585a6134453ae1affb7d85fb2046386 Jul 07 19:39:53.992204 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:53.992187 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:39:54.316475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:54.316442 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" event={"ID":"83e36ef8-a7d3-4189-a987-8e709e49c9e3","Type":"ContainerStarted","Data":"d933a573ee2e2ee48d8f615b045b92ca5585a6134453ae1affb7d85fb2046386"} Jul 07 19:39:55.323983 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:55.323945 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" event={"ID":"83e36ef8-a7d3-4189-a987-8e709e49c9e3","Type":"ContainerStarted","Data":"6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012"} Jul 07 19:39:55.324367 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:55.324013 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:39:55.342557 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:55.342508 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" podStartSLOduration=1.842921147 podStartE2EDuration="2.342493299s" totalImagesPullingTime="499.572152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:39:55.3413648 +0000 UTC m=+1901.276990161" watchObservedRunningTime="2026-07-07 19:39:55.342493299 +0000 UTC m=+1901.278118659" Jul 07 19:39:57.720142 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:57.720104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:58.043231 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:58.043188 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:58.368964 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:58.368858 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:58.819271 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:39:58.819232 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:01.050161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.050132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:01.169571 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.169537 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85"] Jul 07 19:40:01.173780 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.173756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.177193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.177160 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-548bc4-serving-cert\"" Jul 07 19:40:01.188838 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.188813 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85"] Jul 07 19:40:01.199155 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.199119 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:01.257443 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.257408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0452c279-c38e-40b7-9960-7602267b4444-maas-api-tls\") pod \"maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.257610 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.257455 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0452c279-c38e-40b7-9960-7602267b4444-openshift-service-ca\") pod \"maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.257610 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.257566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgpm4\" (UniqueName: \"kubernetes.io/projected/0452c279-c38e-40b7-9960-7602267b4444-kube-api-access-lgpm4\") pod \"maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.335201 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.335113 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:40:01.356823 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.356768 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:01.358106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.358084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0452c279-c38e-40b7-9960-7602267b4444-openshift-service-ca\") pod \"maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.358237 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.358160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgpm4\" (UniqueName: \"kubernetes.io/projected/0452c279-c38e-40b7-9960-7602267b4444-kube-api-access-lgpm4\") pod \"maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.358294 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.358258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0452c279-c38e-40b7-9960-7602267b4444-maas-api-tls\") pod \"maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.358715 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.358696 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0452c279-c38e-40b7-9960-7602267b4444-openshift-service-ca\") pod \"maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.360694 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.360675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0452c279-c38e-40b7-9960-7602267b4444-maas-api-tls\") pod \"maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.373499 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.373475 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgpm4\" (UniqueName: \"kubernetes.io/projected/0452c279-c38e-40b7-9960-7602267b4444-kube-api-access-lgpm4\") pod \"maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.453568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.453533 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:01.485983 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.485947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:01.641284 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.641249 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85"] Jul 07 19:40:01.643355 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:40:01.643329 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0452c279_c38e_40b7_9960_7602267b4444.slice/crio-03bbb9dc77efb596bd22dfb607569529d6fec9007742251116bab78747bbd8ea WatchSource:0}: Error finding container 03bbb9dc77efb596bd22dfb607569529d6fec9007742251116bab78747bbd8ea: Status 404 returned error can't find the container with id 03bbb9dc77efb596bd22dfb607569529d6fec9007742251116bab78747bbd8ea Jul 07 19:40:01.740686 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.740631 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:01.894724 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:01.894608 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:02.361450 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:02.361410 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" event={"ID":"0452c279-c38e-40b7-9960-7602267b4444","Type":"ContainerStarted","Data":"03bbb9dc77efb596bd22dfb607569529d6fec9007742251116bab78747bbd8ea"} Jul 07 19:40:03.368591 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:03.368556 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" event={"ID":"0452c279-c38e-40b7-9960-7602267b4444","Type":"ContainerStarted","Data":"45b2c51c8430b35a5c9c1041798638648b907c90bd55c2883ef4b24349128018"} Jul 07 19:40:03.369032 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:03.368683 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:03.394582 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:03.391679 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" podStartSLOduration=1.580140218 podStartE2EDuration="2.391656503s" totalImagesPullingTime="811.516285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:40:03.389473571 +0000 UTC m=+1909.325098932" watchObservedRunningTime="2026-07-07 19:40:03.391656503 +0000 UTC m=+1909.327281865" Jul 07 19:40:05.490641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:05.490596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:07.866979 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:07.866934 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:08.346040 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:08.345994 2577 patch_prober.go:28] interesting pod/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 19:40:08.346217 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:08.346070 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 19:40:09.379750 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:09.379717 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:40:10.411387 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:10.411357 2577 generic.go:356] "Generic (PLEG): container finished" podID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerID="54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d" exitCode=0 Jul 07 19:40:10.411784 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:10.411399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" event={"ID":"fe231d27-7b7d-4f81-ab6e-b32f2905b028","Type":"ContainerDied","Data":"54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d"} Jul 07 19:40:10.411868 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:10.411852 2577 scope.go:117] "RemoveContainer" containerID="54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d" Jul 07 19:40:10.413032 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:10.413005 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:40:10.413121 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:10.413072 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:40:10.413121 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:10.413103 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:40:11.422760 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:11.422693 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" event={"ID":"fe231d27-7b7d-4f81-ab6e-b32f2905b028","Type":"ContainerStarted","Data":"0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6"} Jul 07 19:40:12.009640 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:12.009603 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:40:12.010020 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:12.009995 2577 patch_prober.go:28] interesting pod/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 07 19:40:12.010138 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:12.010046 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 07 19:40:13.009281 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:13.009241 2577 patch_prober.go:28] interesting pod/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 07 19:40:13.009656 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:13.009313 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 07 19:40:14.009378 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:14.009342 2577 patch_prober.go:28] interesting pod/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 07 19:40:14.009852 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:14.009407 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 07 19:40:15.009153 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:15.009116 2577 patch_prober.go:28] interesting pod/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 07 19:40:15.009332 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:15.009177 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 07 19:40:15.543400 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:15.543362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:15.644533 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:15.644493 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:16.012275 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:16.012250 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:40:16.012526 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:16.012507 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:40:16.013055 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:16.013037 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:40:23.754502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:23.754472 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:25.276470 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:25.276433 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:40.275251 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:40:40.275205 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:14.568106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:14.568062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:15.693641 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:15.693603 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:15.913454 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:15.913415 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:16.340351 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:16.340318 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:16.682816 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:16.682729 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:16.803689 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:16.803656 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:17.091847 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.091815 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj"] Jul 07 19:41:17.092075 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.092047 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" podUID="83e36ef8-a7d3-4189-a987-8e709e49c9e3" containerName="maas-api" containerID="cri-o://6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012" gracePeriod=30 Jul 07 19:41:17.186899 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.186850 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:17.367416 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.367391 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:41:17.535982 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.535952 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e36ef8-a7d3-4189-a987-8e709e49c9e3-maas-api-tls\") pod \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " Jul 07 19:41:17.536144 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.536046 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rb9d\" (UniqueName: \"kubernetes.io/projected/83e36ef8-a7d3-4189-a987-8e709e49c9e3-kube-api-access-9rb9d\") pod \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " Jul 07 19:41:17.536186 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.536157 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e36ef8-a7d3-4189-a987-8e709e49c9e3-openshift-service-ca\") pod \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\" (UID: \"83e36ef8-a7d3-4189-a987-8e709e49c9e3\") " Jul 07 19:41:17.536518 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.536493 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e36ef8-a7d3-4189-a987-8e709e49c9e3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83e36ef8-a7d3-4189-a987-8e709e49c9e3" (UID: "83e36ef8-a7d3-4189-a987-8e709e49c9e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:41:17.538241 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.538215 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e36ef8-a7d3-4189-a987-8e709e49c9e3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83e36ef8-a7d3-4189-a987-8e709e49c9e3" (UID: "83e36ef8-a7d3-4189-a987-8e709e49c9e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:41:17.538325 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.538217 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e36ef8-a7d3-4189-a987-8e709e49c9e3-kube-api-access-9rb9d" (OuterVolumeSpecName: "kube-api-access-9rb9d") pod "83e36ef8-a7d3-4189-a987-8e709e49c9e3" (UID: "83e36ef8-a7d3-4189-a987-8e709e49c9e3"). InnerVolumeSpecName "kube-api-access-9rb9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:41:17.637389 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.637306 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e36ef8-a7d3-4189-a987-8e709e49c9e3-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:41:17.637389 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.637335 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e36ef8-a7d3-4189-a987-8e709e49c9e3-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:41:17.637389 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.637343 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rb9d\" (UniqueName: \"kubernetes.io/projected/83e36ef8-a7d3-4189-a987-8e709e49c9e3-kube-api-access-9rb9d\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:41:17.771083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.771044 2577 generic.go:356] "Generic (PLEG): container finished" podID="83e36ef8-a7d3-4189-a987-8e709e49c9e3" containerID="6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012" exitCode=0 Jul 07 19:41:17.771083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.771085 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" Jul 07 19:41:17.771331 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.771110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" event={"ID":"83e36ef8-a7d3-4189-a987-8e709e49c9e3","Type":"ContainerDied","Data":"6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012"} Jul 07 19:41:17.771331 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.771157 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj" event={"ID":"83e36ef8-a7d3-4189-a987-8e709e49c9e3","Type":"ContainerDied","Data":"d933a573ee2e2ee48d8f615b045b92ca5585a6134453ae1affb7d85fb2046386"} Jul 07 19:41:17.771331 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.771178 2577 scope.go:117] "RemoveContainer" containerID="6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012" Jul 07 19:41:17.782107 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.782088 2577 scope.go:117] "RemoveContainer" containerID="6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012" Jul 07 19:41:17.782366 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:41:17.782347 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012\": container with ID starting with 6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012 not found: ID does not exist" containerID="6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012" Jul 07 19:41:17.782428 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.782373 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012"} err="failed to get container status \"6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012\": rpc error: code = NotFound desc = could not find container \"6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012\": container with ID starting with 6317ca7646822a48cb52b3e670a4a1d8ee3bb8e90d79ea2b7cb8cf4ee30e4012 not found: ID does not exist" Jul 07 19:41:17.822024 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.821960 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj"] Jul 07 19:41:17.836825 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:17.836802 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-444f1d-5b5bcc9c64-mr7qj"] Jul 07 19:41:18.604203 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:18.604171 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e36ef8-a7d3-4189-a987-8e709e49c9e3" path="/var/lib/kubelet/pods/83e36ef8-a7d3-4189-a987-8e709e49c9e3/volumes" Jul 07 19:41:21.392863 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:21.392830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:52.440320 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:41:52.440278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:00.250318 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:00.250277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:00.604682 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:00.604600 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc"] Jul 07 19:42:00.604844 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:00.604775 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" containerID="cri-o://0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6" gracePeriod=30 Jul 07 19:42:00.687599 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:00.687562 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:00.950120 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:00.950023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:01.015063 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:01.015028 2577 patch_prober.go:28] interesting pod/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 19:42:01.015223 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:01.015086 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:42:01.280927 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:01.280897 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:01.799468 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:01.799432 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:02.354462 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:02.354430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:02.558265 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:02.558228 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:02.798961 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:02.798927 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85"] Jul 07 19:42:02.799165 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:02.799140 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" podUID="0452c279-c38e-40b7-9960-7602267b4444" containerName="maas-api" containerID="cri-o://45b2c51c8430b35a5c9c1041798638648b907c90bd55c2883ef4b24349128018" gracePeriod=30 Jul 07 19:42:02.897771 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:02.897732 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:03.006890 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.006839 2577 generic.go:356] "Generic (PLEG): container finished" podID="0452c279-c38e-40b7-9960-7602267b4444" containerID="45b2c51c8430b35a5c9c1041798638648b907c90bd55c2883ef4b24349128018" exitCode=0 Jul 07 19:42:03.007051 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.006914 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" event={"ID":"0452c279-c38e-40b7-9960-7602267b4444","Type":"ContainerDied","Data":"45b2c51c8430b35a5c9c1041798638648b907c90bd55c2883ef4b24349128018"} Jul 07 19:42:03.048792 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.048766 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:42:03.085285 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.085185 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0452c279-c38e-40b7-9960-7602267b4444-maas-api-tls\") pod \"0452c279-c38e-40b7-9960-7602267b4444\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " Jul 07 19:42:03.085285 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.085242 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0452c279-c38e-40b7-9960-7602267b4444-openshift-service-ca\") pod \"0452c279-c38e-40b7-9960-7602267b4444\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " Jul 07 19:42:03.085515 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.085328 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgpm4\" (UniqueName: \"kubernetes.io/projected/0452c279-c38e-40b7-9960-7602267b4444-kube-api-access-lgpm4\") pod \"0452c279-c38e-40b7-9960-7602267b4444\" (UID: \"0452c279-c38e-40b7-9960-7602267b4444\") " Jul 07 19:42:03.085681 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.085650 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0452c279-c38e-40b7-9960-7602267b4444-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0452c279-c38e-40b7-9960-7602267b4444" (UID: "0452c279-c38e-40b7-9960-7602267b4444"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:42:03.087802 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.087773 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0452c279-c38e-40b7-9960-7602267b4444-kube-api-access-lgpm4" (OuterVolumeSpecName: "kube-api-access-lgpm4") pod "0452c279-c38e-40b7-9960-7602267b4444" (UID: "0452c279-c38e-40b7-9960-7602267b4444"). InnerVolumeSpecName "kube-api-access-lgpm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:42:03.088066 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.088043 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0452c279-c38e-40b7-9960-7602267b4444-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0452c279-c38e-40b7-9960-7602267b4444" (UID: "0452c279-c38e-40b7-9960-7602267b4444"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:42:03.186826 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.186791 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0452c279-c38e-40b7-9960-7602267b4444-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:03.186826 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.186822 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0452c279-c38e-40b7-9960-7602267b4444-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:03.187043 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:03.186836 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgpm4\" (UniqueName: \"kubernetes.io/projected/0452c279-c38e-40b7-9960-7602267b4444-kube-api-access-lgpm4\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:04.014394 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:04.014364 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" Jul 07 19:42:04.014394 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:04.014381 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85" event={"ID":"0452c279-c38e-40b7-9960-7602267b4444","Type":"ContainerDied","Data":"03bbb9dc77efb596bd22dfb607569529d6fec9007742251116bab78747bbd8ea"} Jul 07 19:42:04.014908 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:04.014436 2577 scope.go:117] "RemoveContainer" containerID="45b2c51c8430b35a5c9c1041798638648b907c90bd55c2883ef4b24349128018" Jul 07 19:42:04.084056 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:04.084023 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85"] Jul 07 19:42:04.098141 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:04.098113 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-548bc4-57fb7678dd-ltz85"] Jul 07 19:42:04.605085 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:04.605056 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0452c279-c38e-40b7-9960-7602267b4444" path="/var/lib/kubelet/pods/0452c279-c38e-40b7-9960-7602267b4444/volumes" Jul 07 19:42:05.848988 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.848958 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:42:05.913991 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.913919 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-socket\") pod \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " Jul 07 19:42:05.913991 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.913970 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-podinfo\") pod \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " Jul 07 19:42:05.914380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914009 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-credential-socket\") pod \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " Jul 07 19:42:05.914380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914029 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-token\") pod \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " Jul 07 19:42:05.914380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914047 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istiod-ca-cert\") pod \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " Jul 07 19:42:05.914380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914132 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-data\") pod \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " Jul 07 19:42:05.914380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914172 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d8jf\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-kube-api-access-9d8jf\") pod \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " Jul 07 19:42:05.914380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914208 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe231d27-7b7d-4f81-ab6e-b32f2905b028" (UID: "fe231d27-7b7d-4f81-ab6e-b32f2905b028"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:05.914380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914245 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-envoy\") pod \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " Jul 07 19:42:05.914380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914276 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-certs\") pod \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\" (UID: \"fe231d27-7b7d-4f81-ab6e-b32f2905b028\") " Jul 07 19:42:05.914380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914321 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe231d27-7b7d-4f81-ab6e-b32f2905b028" (UID: "fe231d27-7b7d-4f81-ab6e-b32f2905b028"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:05.914858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914537 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe231d27-7b7d-4f81-ab6e-b32f2905b028" (UID: "fe231d27-7b7d-4f81-ab6e-b32f2905b028"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:42:05.914858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914666 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:05.914858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914754 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe231d27-7b7d-4f81-ab6e-b32f2905b028" (UID: "fe231d27-7b7d-4f81-ab6e-b32f2905b028"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:05.914858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914684 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-credential-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:05.914858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.914785 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istiod-ca-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:05.916159 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.916135 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe231d27-7b7d-4f81-ab6e-b32f2905b028" (UID: "fe231d27-7b7d-4f81-ab6e-b32f2905b028"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:05.917025 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.916966 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-kube-api-access-9d8jf" (OuterVolumeSpecName: "kube-api-access-9d8jf") pod "fe231d27-7b7d-4f81-ab6e-b32f2905b028" (UID: "fe231d27-7b7d-4f81-ab6e-b32f2905b028"). InnerVolumeSpecName "kube-api-access-9d8jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:42:05.917285 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.917259 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe231d27-7b7d-4f81-ab6e-b32f2905b028" (UID: "fe231d27-7b7d-4f81-ab6e-b32f2905b028"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:42:05.917534 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.917509 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe231d27-7b7d-4f81-ab6e-b32f2905b028" (UID: "fe231d27-7b7d-4f81-ab6e-b32f2905b028"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:05.917534 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:05.917519 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe231d27-7b7d-4f81-ab6e-b32f2905b028" (UID: "fe231d27-7b7d-4f81-ab6e-b32f2905b028"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:42:06.016254 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.016225 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-podinfo\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:06.016254 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.016253 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-token\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:06.016254 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.016262 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-data\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:06.016464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.016269 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d8jf\" (UniqueName: \"kubernetes.io/projected/fe231d27-7b7d-4f81-ab6e-b32f2905b028-kube-api-access-9d8jf\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:06.016464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.016282 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-istio-envoy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:06.016464 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.016292 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe231d27-7b7d-4f81-ab6e-b32f2905b028-workload-certs\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:06.028548 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.028521 2577 generic.go:356] "Generic (PLEG): container finished" podID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerID="0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6" exitCode=0 Jul 07 19:42:06.028703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.028577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" event={"ID":"fe231d27-7b7d-4f81-ab6e-b32f2905b028","Type":"ContainerDied","Data":"0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6"} Jul 07 19:42:06.028703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.028585 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" Jul 07 19:42:06.028703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.028621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc" event={"ID":"fe231d27-7b7d-4f81-ab6e-b32f2905b028","Type":"ContainerDied","Data":"3ebadae10cdee16d63518c9da5ae61a611ba06ccec731787e9b44622db260fee"} Jul 07 19:42:06.028703 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.028651 2577 scope.go:117] "RemoveContainer" containerID="0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6" Jul 07 19:42:06.038832 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.038812 2577 scope.go:117] "RemoveContainer" containerID="54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d" Jul 07 19:42:06.046848 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.046826 2577 scope.go:117] "RemoveContainer" containerID="0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6" Jul 07 19:42:06.047129 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:42:06.047106 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6\": container with ID starting with 0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6 not found: ID does not exist" containerID="0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6" Jul 07 19:42:06.047182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.047138 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6"} err="failed to get container status \"0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6\": rpc error: code = NotFound desc = could not find container \"0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6\": container with ID starting with 0f3f069dbab04d08af7bece95eeaa32e17ca4b0306695281e37f016f6f9f94f6 not found: ID does not exist" Jul 07 19:42:06.047182 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.047154 2577 scope.go:117] "RemoveContainer" containerID="54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d" Jul 07 19:42:06.047412 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:42:06.047393 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d\": container with ID starting with 54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d not found: ID does not exist" containerID="54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d" Jul 07 19:42:06.047472 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.047422 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d"} err="failed to get container status \"54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d\": rpc error: code = NotFound desc = could not find container \"54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d\": container with ID starting with 54d084a97287b0f1a9a2d1e0bdbe9db635c0e94b6aeed48ee67e3e27e12f460d not found: ID does not exist" Jul 07 19:42:06.077805 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.077784 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc"] Jul 07 19:42:06.082611 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.082590 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-444f1d-openshift-default-55645ddb8c-4ltzc"] Jul 07 19:42:06.254204 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.254171 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:06.448590 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.448546 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:06.604505 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:06.604431 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" path="/var/lib/kubelet/pods/fe231d27-7b7d-4f81-ab6e-b32f2905b028/volumes" Jul 07 19:42:10.095436 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:10.095402 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:10.331557 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:10.331523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:10.589821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:10.589788 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:10.891177 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:10.891082 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:11.155157 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.155069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:11.636658 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.636629 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx"] Jul 07 19:42:11.637068 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637054 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" Jul 07 19:42:11.637125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637070 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" Jul 07 19:42:11.637125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637078 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0452c279-c38e-40b7-9960-7602267b4444" containerName="maas-api" Jul 07 19:42:11.637125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637083 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0452c279-c38e-40b7-9960-7602267b4444" containerName="maas-api" Jul 07 19:42:11.637125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637089 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" Jul 07 19:42:11.637125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637095 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" Jul 07 19:42:11.637125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637113 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e36ef8-a7d3-4189-a987-8e709e49c9e3" containerName="maas-api" Jul 07 19:42:11.637125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637119 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e36ef8-a7d3-4189-a987-8e709e49c9e3" containerName="maas-api" Jul 07 19:42:11.637345 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637184 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" Jul 07 19:42:11.637345 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637193 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe231d27-7b7d-4f81-ab6e-b32f2905b028" containerName="istio-proxy" Jul 07 19:42:11.637345 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637203 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0452c279-c38e-40b7-9960-7602267b4444" containerName="maas-api" Jul 07 19:42:11.637345 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.637208 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e36ef8-a7d3-4189-a987-8e709e49c9e3" containerName="maas-api" Jul 07 19:42:11.641995 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.641974 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.651397 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.651374 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f047a3-openshift-default-dockercfg-288x2\"" Jul 07 19:42:11.678453 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.678421 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx"] Jul 07 19:42:11.708974 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.708946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:11.768154 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.768125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-data\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.768289 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.768158 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-podinfo\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.768289 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.768179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-credential-socket\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.768289 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.768196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-envoy\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.768399 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.768308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istiod-ca-cert\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.768399 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.768344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-socket\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.768399 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.768364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-token\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.768492 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.768409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-certs\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.768492 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.768454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lv2k\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-kube-api-access-9lv2k\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.869488 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869448 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lv2k\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-kube-api-access-9lv2k\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.869627 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-data\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.869627 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-podinfo\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.869627 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-credential-socket\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.869627 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-envoy\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.869869 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istiod-ca-cert\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.869869 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-socket\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.869869 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-token\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.869869 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-certs\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.870088 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.869956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-credential-socket\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.870140 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.870093 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-data\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.870202 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.870182 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-socket\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.870291 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.870271 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-certs\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.870511 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.870343 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istiod-ca-cert\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.872070 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.872049 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-envoy\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.872207 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.872191 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-podinfo\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.887445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.887382 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lv2k\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-kube-api-access-9lv2k\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.891362 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.891336 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-token\") pod \"e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:11.954714 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:11.954690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:12.147349 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:12.147259 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx"] Jul 07 19:42:12.150403 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:42:12.150377 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35579d2e_cdc9_4284_acbe_122ec7bc7d63.slice/crio-8c7c87b7c37f017e6b8e64b20d0693d8a48b97e9d4d6d56969f24acd8fa25d1d WatchSource:0}: Error finding container 8c7c87b7c37f017e6b8e64b20d0693d8a48b97e9d4d6d56969f24acd8fa25d1d: Status 404 returned error can't find the container with id 8c7c87b7c37f017e6b8e64b20d0693d8a48b97e9d4d6d56969f24acd8fa25d1d Jul 07 19:42:12.152619 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:12.152581 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:42:12.152699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:12.152664 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:42:12.152699 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:12.152691 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:42:12.188453 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:12.188419 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:13.070602 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:13.070568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" event={"ID":"35579d2e-cdc9-4284-acbe-122ec7bc7d63","Type":"ContainerStarted","Data":"33106ef9220e2440135af26ed891c1ebc7db846d5e618a47e0530c943e288010"} Jul 07 19:42:13.070768 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:13.070619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" event={"ID":"35579d2e-cdc9-4284-acbe-122ec7bc7d63","Type":"ContainerStarted","Data":"8c7c87b7c37f017e6b8e64b20d0693d8a48b97e9d4d6d56969f24acd8fa25d1d"} Jul 07 19:42:13.106674 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:13.106630 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" podStartSLOduration=2.106614132 podStartE2EDuration="2.106614132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:42:13.105535065 +0000 UTC m=+2039.041160427" watchObservedRunningTime="2026-07-07 19:42:13.106614132 +0000 UTC m=+2039.042239547" Jul 07 19:42:13.955136 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:13.955106 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:13.959993 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:13.959969 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:14.076279 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:14.076250 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:14.077209 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:14.077192 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:15.344495 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.344457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:15.647551 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.647468 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg"] Jul 07 19:42:15.652601 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.652580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.659737 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.659716 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-f047a3-serving-cert\"" Jul 07 19:42:15.665057 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.665034 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:15.692064 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.692036 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg"] Jul 07 19:42:15.810351 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.810317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4z5b\" (UniqueName: \"kubernetes.io/projected/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-kube-api-access-d4z5b\") pod \"maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.810351 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.810354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-maas-api-tls\") pod \"maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.810579 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.810377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-openshift-service-ca\") pod \"maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.912011 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.911928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-openshift-service-ca\") pod \"maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.912175 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.912055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4z5b\" (UniqueName: \"kubernetes.io/projected/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-kube-api-access-d4z5b\") pod \"maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.912175 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.912085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-maas-api-tls\") pod \"maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.912590 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.912570 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-openshift-service-ca\") pod \"maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.914537 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.914520 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-maas-api-tls\") pod \"maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.942672 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.942638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:15.949269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.949241 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4z5b\" (UniqueName: \"kubernetes.io/projected/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-kube-api-access-d4z5b\") pod \"maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:15.962992 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:15.962968 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:16.123307 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:16.123249 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg"] Jul 07 19:42:16.125634 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:42:16.125608 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bcd87a_2492_4548_a0e2_1fc1b31f76f8.slice/crio-0cec2028b7a541280299ec156fe36bc1acc29eeff92ff19b6418eb028b892ed6 WatchSource:0}: Error finding container 0cec2028b7a541280299ec156fe36bc1acc29eeff92ff19b6418eb028b892ed6: Status 404 returned error can't find the container with id 0cec2028b7a541280299ec156fe36bc1acc29eeff92ff19b6418eb028b892ed6 Jul 07 19:42:16.131719 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:16.131678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:16.772482 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:16.772443 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:17.094865 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:17.094826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" event={"ID":"17bcd87a-2492-4548-a0e2-1fc1b31f76f8","Type":"ContainerStarted","Data":"46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f"} Jul 07 19:42:17.095097 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:17.094905 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" event={"ID":"17bcd87a-2492-4548-a0e2-1fc1b31f76f8","Type":"ContainerStarted","Data":"0cec2028b7a541280299ec156fe36bc1acc29eeff92ff19b6418eb028b892ed6"} Jul 07 19:42:17.095097 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:17.094933 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:17.103445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:17.103417 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:17.170100 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:17.170058 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" podStartSLOduration=1.75740457 podStartE2EDuration="2.170041846s" totalImagesPullingTime="412.637276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:42:17.146559615 +0000 UTC m=+2043.082184975" watchObservedRunningTime="2026-07-07 19:42:17.170041846 +0000 UTC m=+2043.105667206" Jul 07 19:42:19.951125 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:19.951089 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:20.479145 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:20.479106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:20.907169 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:20.907132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:21.329911 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:21.329847 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:23.108866 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:23.108832 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:24.087488 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.087449 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:24.208924 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.208821 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:24.217624 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.217594 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9"] Jul 07 19:42:24.221935 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.221916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.224795 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.224775 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-bdaaac-serving-cert\"" Jul 07 19:42:24.238069 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.238047 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9"] Jul 07 19:42:24.352283 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.352189 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:24.407264 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.407218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg46b\" (UniqueName: \"kubernetes.io/projected/0446a60d-e056-4885-a249-eb7fb40d3e5d-kube-api-access-cg46b\") pod \"maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.407467 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.407279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0446a60d-e056-4885-a249-eb7fb40d3e5d-maas-api-tls\") pod \"maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.407467 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.407309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0446a60d-e056-4885-a249-eb7fb40d3e5d-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.474745 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.474704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:24.508040 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.508007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg46b\" (UniqueName: \"kubernetes.io/projected/0446a60d-e056-4885-a249-eb7fb40d3e5d-kube-api-access-cg46b\") pod \"maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.508197 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.508048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0446a60d-e056-4885-a249-eb7fb40d3e5d-maas-api-tls\") pod \"maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.508197 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.508066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0446a60d-e056-4885-a249-eb7fb40d3e5d-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.508797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.508778 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0446a60d-e056-4885-a249-eb7fb40d3e5d-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.510670 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.510651 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0446a60d-e056-4885-a249-eb7fb40d3e5d-maas-api-tls\") pod \"maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.528604 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.528583 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg46b\" (UniqueName: \"kubernetes.io/projected/0446a60d-e056-4885-a249-eb7fb40d3e5d-kube-api-access-cg46b\") pod \"maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.533449 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.533434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:24.719500 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:24.719474 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9"] Jul 07 19:42:24.721767 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:42:24.721742 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0446a60d_e056_4885_a249_eb7fb40d3e5d.slice/crio-b10da5e8cd0c281c26b453090acf75fc8b47b9003f6a8d72362ab4c1ebe897f4 WatchSource:0}: Error finding container b10da5e8cd0c281c26b453090acf75fc8b47b9003f6a8d72362ab4c1ebe897f4: Status 404 returned error can't find the container with id b10da5e8cd0c281c26b453090acf75fc8b47b9003f6a8d72362ab4c1ebe897f4 Jul 07 19:42:25.138650 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:25.138624 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" event={"ID":"0446a60d-e056-4885-a249-eb7fb40d3e5d","Type":"ContainerStarted","Data":"036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3"} Jul 07 19:42:25.138783 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:25.138673 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" event={"ID":"0446a60d-e056-4885-a249-eb7fb40d3e5d","Type":"ContainerStarted","Data":"b10da5e8cd0c281c26b453090acf75fc8b47b9003f6a8d72362ab4c1ebe897f4"} Jul 07 19:42:25.138783 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:25.138766 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:25.161161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:25.161135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:25.168970 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:25.168930 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" podStartSLOduration=0.829885688 podStartE2EDuration="1.16891635s" totalImagesPullingTime="339.030662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:42:25.166687152 +0000 UTC m=+2051.102312512" watchObservedRunningTime="2026-07-07 19:42:25.16891635 +0000 UTC m=+2051.104541710" Jul 07 19:42:25.263944 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:25.263904 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:28.157121 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:28.157087 2577 generic.go:356] "Generic (PLEG): container finished" podID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerID="33106ef9220e2440135af26ed891c1ebc7db846d5e618a47e0530c943e288010" exitCode=0 Jul 07 19:42:28.157121 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:28.157124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" event={"ID":"35579d2e-cdc9-4284-acbe-122ec7bc7d63","Type":"ContainerDied","Data":"33106ef9220e2440135af26ed891c1ebc7db846d5e618a47e0530c943e288010"} Jul 07 19:42:28.157655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:28.157531 2577 scope.go:117] "RemoveContainer" containerID="33106ef9220e2440135af26ed891c1ebc7db846d5e618a47e0530c943e288010" Jul 07 19:42:28.158455 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:28.158424 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:42:28.158516 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:28.158492 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:42:28.158566 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:28.158523 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:42:28.651602 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:28.651574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:29.076736 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:29.076702 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:29.168306 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:29.168266 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" event={"ID":"35579d2e-cdc9-4284-acbe-122ec7bc7d63","Type":"ContainerStarted","Data":"298f47751079395a2f625acebc49bb467c586b9bea91c6304d9c50c711e426a4"} Jul 07 19:42:29.955077 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:29.955043 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:29.956308 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:29.956280 2577 patch_prober.go:28] interesting pod/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 07 19:42:29.956425 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:29.956329 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 07 19:42:30.955348 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:30.955311 2577 patch_prober.go:28] interesting pod/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 07 19:42:30.955721 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:30.955365 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 07 19:42:31.150932 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:31.150903 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:31.955655 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:31.955624 2577 patch_prober.go:28] interesting pod/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 07 19:42:31.956056 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:31.955675 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 07 19:42:32.955471 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:32.955430 2577 patch_prober.go:28] interesting pod/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 07 19:42:32.955649 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:32.955497 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 07 19:42:33.959696 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:33.959668 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:33.960149 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:33.959952 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:33.960621 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:33.960604 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:37.414855 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:37.414772 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:37.819547 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:37.819510 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:38.083204 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.083132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:38.216317 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.216284 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:38.658863 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.658829 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg"] Jul 07 19:42:38.659268 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.659059 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" podUID="17bcd87a-2492-4548-a0e2-1fc1b31f76f8" containerName="maas-api" containerID="cri-o://46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f" gracePeriod=30 Jul 07 19:42:38.820147 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.820111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:38.920543 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.920478 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:38.964156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.964127 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4z5b\" (UniqueName: \"kubernetes.io/projected/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-kube-api-access-d4z5b\") pod \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " Jul 07 19:42:38.964290 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.964217 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-maas-api-tls\") pod \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " Jul 07 19:42:38.964290 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.964279 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-openshift-service-ca\") pod \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\" (UID: \"17bcd87a-2492-4548-a0e2-1fc1b31f76f8\") " Jul 07 19:42:38.964599 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.964576 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17bcd87a-2492-4548-a0e2-1fc1b31f76f8" (UID: "17bcd87a-2492-4548-a0e2-1fc1b31f76f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:42:38.966229 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.966205 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-kube-api-access-d4z5b" (OuterVolumeSpecName: "kube-api-access-d4z5b") pod "17bcd87a-2492-4548-a0e2-1fc1b31f76f8" (UID: "17bcd87a-2492-4548-a0e2-1fc1b31f76f8"). InnerVolumeSpecName "kube-api-access-d4z5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:42:38.966297 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:38.966265 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17bcd87a-2492-4548-a0e2-1fc1b31f76f8" (UID: "17bcd87a-2492-4548-a0e2-1fc1b31f76f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:42:39.065519 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.065487 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:39.065519 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.065515 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4z5b\" (UniqueName: \"kubernetes.io/projected/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-kube-api-access-d4z5b\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:39.065689 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.065527 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17bcd87a-2492-4548-a0e2-1fc1b31f76f8-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:39.229569 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.229492 2577 generic.go:356] "Generic (PLEG): container finished" podID="17bcd87a-2492-4548-a0e2-1fc1b31f76f8" containerID="46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f" exitCode=0 Jul 07 19:42:39.229569 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.229530 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" event={"ID":"17bcd87a-2492-4548-a0e2-1fc1b31f76f8","Type":"ContainerDied","Data":"46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f"} Jul 07 19:42:39.229569 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.229540 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" Jul 07 19:42:39.229802 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.229598 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg" event={"ID":"17bcd87a-2492-4548-a0e2-1fc1b31f76f8","Type":"ContainerDied","Data":"0cec2028b7a541280299ec156fe36bc1acc29eeff92ff19b6418eb028b892ed6"} Jul 07 19:42:39.229802 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.229621 2577 scope.go:117] "RemoveContainer" containerID="46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f" Jul 07 19:42:39.239476 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.239458 2577 scope.go:117] "RemoveContainer" containerID="46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f" Jul 07 19:42:39.239749 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:42:39.239730 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f\": container with ID starting with 46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f not found: ID does not exist" containerID="46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f" Jul 07 19:42:39.239803 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.239758 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f"} err="failed to get container status \"46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f\": rpc error: code = NotFound desc = could not find container \"46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f\": container with ID starting with 46d3d4c47efdf2540146de3da5c78027e451d1d5bb52f4cd99e13345df85e37f not found: ID does not exist" Jul 07 19:42:39.270373 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.270347 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg"] Jul 07 19:42:39.277495 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:39.277475 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f047a3-69cb9cb4b7-sxzdg"] Jul 07 19:42:40.605246 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:40.605214 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17bcd87a-2492-4548-a0e2-1fc1b31f76f8" path="/var/lib/kubelet/pods/17bcd87a-2492-4548-a0e2-1fc1b31f76f8/volumes" Jul 07 19:42:44.722535 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:44.722503 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:44.823936 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:44.823896 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:45.041253 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:45.041225 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx"] Jul 07 19:42:45.041454 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:45.041432 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" containerID="cri-o://298f47751079395a2f625acebc49bb467c586b9bea91c6304d9c50c711e426a4" gracePeriod=30 Jul 07 19:42:45.060768 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:45.060735 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:45.367711 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:45.367623 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:45.609340 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:45.609302 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:46.026629 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.026590 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:46.127316 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.127283 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:46.228151 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.228120 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:46.362280 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.362201 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9"] Jul 07 19:42:46.362423 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.362400 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" podUID="0446a60d-e056-4885-a249-eb7fb40d3e5d" containerName="maas-api" containerID="cri-o://036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3" gracePeriod=30 Jul 07 19:42:46.423809 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.423775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:46.614365 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.614303 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:46.746605 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.746569 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg46b\" (UniqueName: \"kubernetes.io/projected/0446a60d-e056-4885-a249-eb7fb40d3e5d-kube-api-access-cg46b\") pod \"0446a60d-e056-4885-a249-eb7fb40d3e5d\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " Jul 07 19:42:46.746756 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.746631 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0446a60d-e056-4885-a249-eb7fb40d3e5d-maas-api-tls\") pod \"0446a60d-e056-4885-a249-eb7fb40d3e5d\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " Jul 07 19:42:46.746801 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.746749 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0446a60d-e056-4885-a249-eb7fb40d3e5d-openshift-service-ca\") pod \"0446a60d-e056-4885-a249-eb7fb40d3e5d\" (UID: \"0446a60d-e056-4885-a249-eb7fb40d3e5d\") " Jul 07 19:42:46.747154 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.747100 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0446a60d-e056-4885-a249-eb7fb40d3e5d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0446a60d-e056-4885-a249-eb7fb40d3e5d" (UID: "0446a60d-e056-4885-a249-eb7fb40d3e5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:42:46.747777 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.747380 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0446a60d-e056-4885-a249-eb7fb40d3e5d-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:46.748779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.748748 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0446a60d-e056-4885-a249-eb7fb40d3e5d-kube-api-access-cg46b" (OuterVolumeSpecName: "kube-api-access-cg46b") pod "0446a60d-e056-4885-a249-eb7fb40d3e5d" (UID: "0446a60d-e056-4885-a249-eb7fb40d3e5d"). InnerVolumeSpecName "kube-api-access-cg46b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:42:46.748901 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.748843 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0446a60d-e056-4885-a249-eb7fb40d3e5d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0446a60d-e056-4885-a249-eb7fb40d3e5d" (UID: "0446a60d-e056-4885-a249-eb7fb40d3e5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:42:46.848501 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.848466 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg46b\" (UniqueName: \"kubernetes.io/projected/0446a60d-e056-4885-a249-eb7fb40d3e5d-kube-api-access-cg46b\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:46.848501 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:46.848495 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0446a60d-e056-4885-a249-eb7fb40d3e5d-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:47.273445 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:47.273414 2577 generic.go:356] "Generic (PLEG): container finished" podID="0446a60d-e056-4885-a249-eb7fb40d3e5d" containerID="036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3" exitCode=0 Jul 07 19:42:47.273858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:47.273457 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" Jul 07 19:42:47.273858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:47.273496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" event={"ID":"0446a60d-e056-4885-a249-eb7fb40d3e5d","Type":"ContainerDied","Data":"036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3"} Jul 07 19:42:47.273858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:47.273547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9" event={"ID":"0446a60d-e056-4885-a249-eb7fb40d3e5d","Type":"ContainerDied","Data":"b10da5e8cd0c281c26b453090acf75fc8b47b9003f6a8d72362ab4c1ebe897f4"} Jul 07 19:42:47.273858 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:47.273580 2577 scope.go:117] "RemoveContainer" containerID="036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3" Jul 07 19:42:47.284991 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:47.284973 2577 scope.go:117] "RemoveContainer" containerID="036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3" Jul 07 19:42:47.285235 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:42:47.285217 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3\": container with ID starting with 036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3 not found: ID does not exist" containerID="036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3" Jul 07 19:42:47.285291 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:47.285243 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3"} err="failed to get container status \"036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3\": rpc error: code = NotFound desc = could not find container \"036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3\": container with ID starting with 036147b966e8a99fc4553ae49d9ba8c7bbfcbfeac7b3661d43a942af62b3caa3 not found: ID does not exist" Jul 07 19:42:47.305974 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:47.305950 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9"] Jul 07 19:42:47.314818 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:47.314799 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-bdaaac-dd669fbff-k7dg9"] Jul 07 19:42:48.604174 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:48.604143 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0446a60d-e056-4885-a249-eb7fb40d3e5d" path="/var/lib/kubelet/pods/0446a60d-e056-4885-a249-eb7fb40d3e5d/volumes" Jul 07 19:42:48.962596 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:48.962515 2577 patch_prober.go:28] interesting pod/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 19:42:48.962596 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:48.962577 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:42:50.299930 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.299900 2577 generic.go:356] "Generic (PLEG): container finished" podID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerID="298f47751079395a2f625acebc49bb467c586b9bea91c6304d9c50c711e426a4" exitCode=0 Jul 07 19:42:50.300281 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.299961 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" event={"ID":"35579d2e-cdc9-4284-acbe-122ec7bc7d63","Type":"ContainerDied","Data":"298f47751079395a2f625acebc49bb467c586b9bea91c6304d9c50c711e426a4"} Jul 07 19:42:50.300281 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.300045 2577 scope.go:117] "RemoveContainer" containerID="33106ef9220e2440135af26ed891c1ebc7db846d5e618a47e0530c943e288010" Jul 07 19:42:50.321654 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:42:50.321614 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33106ef9220e2440135af26ed891c1ebc7db846d5e618a47e0530c943e288010\": container with ID starting with 33106ef9220e2440135af26ed891c1ebc7db846d5e618a47e0530c943e288010 not found: ID does not exist" containerID="33106ef9220e2440135af26ed891c1ebc7db846d5e618a47e0530c943e288010" Jul 07 19:42:50.323470 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.323452 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:50.487366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.487330 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-podinfo\") pod \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " Jul 07 19:42:50.487366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.487363 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-envoy\") pod \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " Jul 07 19:42:50.487612 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.487413 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-token\") pod \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " Jul 07 19:42:50.487612 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.487474 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-certs\") pod \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " Jul 07 19:42:50.487612 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.487517 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istiod-ca-cert\") pod \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " Jul 07 19:42:50.487612 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.487601 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lv2k\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-kube-api-access-9lv2k\") pod \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " Jul 07 19:42:50.487828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.487646 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-socket\") pod \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " Jul 07 19:42:50.487828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.487677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-data\") pod \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " Jul 07 19:42:50.487828 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.487736 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-credential-socket\") pod \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\" (UID: \"35579d2e-cdc9-4284-acbe-122ec7bc7d63\") " Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.488116 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35579d2e-cdc9-4284-acbe-122ec7bc7d63" (UID: "35579d2e-cdc9-4284-acbe-122ec7bc7d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.488337 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35579d2e-cdc9-4284-acbe-122ec7bc7d63" (UID: "35579d2e-cdc9-4284-acbe-122ec7bc7d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.488538 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35579d2e-cdc9-4284-acbe-122ec7bc7d63" (UID: "35579d2e-cdc9-4284-acbe-122ec7bc7d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.488800 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35579d2e-cdc9-4284-acbe-122ec7bc7d63" (UID: "35579d2e-cdc9-4284-acbe-122ec7bc7d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.488940 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-data" (OuterVolumeSpecName: "istio-data") pod "35579d2e-cdc9-4284-acbe-122ec7bc7d63" (UID: "35579d2e-cdc9-4284-acbe-122ec7bc7d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.489080 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-certs\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.489099 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istiod-ca-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.489111 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-workload-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.489123 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-data\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:50.489161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.489135 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-credential-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:50.490082 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.490055 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-token" (OuterVolumeSpecName: "istio-token") pod "35579d2e-cdc9-4284-acbe-122ec7bc7d63" (UID: "35579d2e-cdc9-4284-acbe-122ec7bc7d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:42:50.490228 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.490202 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-kube-api-access-9lv2k" (OuterVolumeSpecName: "kube-api-access-9lv2k") pod "35579d2e-cdc9-4284-acbe-122ec7bc7d63" (UID: "35579d2e-cdc9-4284-acbe-122ec7bc7d63"). InnerVolumeSpecName "kube-api-access-9lv2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:42:50.490296 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.490237 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35579d2e-cdc9-4284-acbe-122ec7bc7d63" (UID: "35579d2e-cdc9-4284-acbe-122ec7bc7d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:50.490371 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.490357 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35579d2e-cdc9-4284-acbe-122ec7bc7d63" (UID: "35579d2e-cdc9-4284-acbe-122ec7bc7d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:42:50.590213 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.590128 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-podinfo\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:50.590213 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.590156 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-envoy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:50.590213 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.590164 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-istio-token\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:50.590213 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:50.590174 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lv2k\" (UniqueName: \"kubernetes.io/projected/35579d2e-cdc9-4284-acbe-122ec7bc7d63-kube-api-access-9lv2k\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:42:51.305919 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:51.305887 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" Jul 07 19:42:51.305919 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:51.305904 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx" event={"ID":"35579d2e-cdc9-4284-acbe-122ec7bc7d63","Type":"ContainerDied","Data":"8c7c87b7c37f017e6b8e64b20d0693d8a48b97e9d4d6d56969f24acd8fa25d1d"} Jul 07 19:42:51.306337 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:51.305964 2577 scope.go:117] "RemoveContainer" containerID="298f47751079395a2f625acebc49bb467c586b9bea91c6304d9c50c711e426a4" Jul 07 19:42:51.328030 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:51.328003 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx"] Jul 07 19:42:51.335304 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:51.335282 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f047a3-openshift-default-6648f7c96f-b5fxx"] Jul 07 19:42:52.604582 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:52.604544 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" path="/var/lib/kubelet/pods/35579d2e-cdc9-4284-acbe-122ec7bc7d63/volumes" Jul 07 19:42:53.054323 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:53.054286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:53.130424 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:53.130396 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:53.469787 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:53.469683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:53.620432 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:53.620397 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:53.997591 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:53.997560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:54.374156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374074 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5"] Jul 07 19:42:54.374572 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374554 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" Jul 07 19:42:54.374663 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374576 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" Jul 07 19:42:54.374663 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374596 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0446a60d-e056-4885-a249-eb7fb40d3e5d" containerName="maas-api" Jul 07 19:42:54.374663 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374604 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0446a60d-e056-4885-a249-eb7fb40d3e5d" containerName="maas-api" Jul 07 19:42:54.374663 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374635 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" Jul 07 19:42:54.374663 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374643 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" Jul 07 19:42:54.374663 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374653 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17bcd87a-2492-4548-a0e2-1fc1b31f76f8" containerName="maas-api" Jul 07 19:42:54.374663 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374661 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17bcd87a-2492-4548-a0e2-1fc1b31f76f8" containerName="maas-api" Jul 07 19:42:54.375046 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374762 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0446a60d-e056-4885-a249-eb7fb40d3e5d" containerName="maas-api" Jul 07 19:42:54.375046 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374775 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" Jul 07 19:42:54.375046 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374786 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17bcd87a-2492-4548-a0e2-1fc1b31f76f8" containerName="maas-api" Jul 07 19:42:54.375046 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.374796 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="35579d2e-cdc9-4284-acbe-122ec7bc7d63" containerName="istio-proxy" Jul 07 19:42:54.378411 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.378391 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.388978 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.388960 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-778336-openshift-default-dockercfg-zrkdm\"" Jul 07 19:42:54.400833 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.400813 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5"] Jul 07 19:42:54.451551 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.451522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:54.526532 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.526501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbz67\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-kube-api-access-vbz67\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.526691 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.526545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-certs\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.526691 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.526567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-istio-token\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.526691 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.526656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-socket\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.526691 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.526684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-envoy\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.526848 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.526716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687df1f5-270f-4afe-afb5-472c0addf82f-istio-podinfo\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.526848 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.526753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-credential-socket\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.526848 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.526793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-data\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.526848 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.526829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687df1f5-270f-4afe-afb5-472c0addf82f-istiod-ca-cert\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.608900 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.608853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:54.627546 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.627457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-certs\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628005 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.627574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-istio-token\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628005 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.627643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-socket\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628005 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.627892 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-certs\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628005 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.627946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-envoy\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628005 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.627974 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-socket\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628005 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.627987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687df1f5-270f-4afe-afb5-472c0addf82f-istio-podinfo\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628346 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.628010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-credential-socket\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628346 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.628038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-data\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628346 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.628089 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687df1f5-270f-4afe-afb5-472c0addf82f-istiod-ca-cert\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628346 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.628153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbz67\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-kube-api-access-vbz67\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628546 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.628389 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-credential-socket\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.628780 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.628750 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-data\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.629012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.628983 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687df1f5-270f-4afe-afb5-472c0addf82f-istiod-ca-cert\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.631048 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.631025 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-envoy\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.631256 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.631233 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687df1f5-270f-4afe-afb5-472c0addf82f-istio-podinfo\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.643403 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.643382 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-istio-token\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.643490 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.643385 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbz67\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-kube-api-access-vbz67\") pod \"e2e-shared-a-778336-openshift-default-f645995dc-xlxc5\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.688436 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.688413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:54.866267 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.866240 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5"] Jul 07 19:42:54.869673 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:42:54.869645 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687df1f5_270f_4afe_afb5_472c0addf82f.slice/crio-8d2c9f56f760f5cb8845ea42e703b15d4da6cad94e1ae98c38950b8114801bfe WatchSource:0}: Error finding container 8d2c9f56f760f5cb8845ea42e703b15d4da6cad94e1ae98c38950b8114801bfe: Status 404 returned error can't find the container with id 8d2c9f56f760f5cb8845ea42e703b15d4da6cad94e1ae98c38950b8114801bfe Jul 07 19:42:54.871948 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.871913 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:42:54.872048 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.871979 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:42:54.872048 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:54.872026 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:42:55.330243 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:55.330203 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" event={"ID":"687df1f5-270f-4afe-afb5-472c0addf82f","Type":"ContainerStarted","Data":"cffd0bc89acc063daef36c736ae3111b221e91d00043d314d164f79f2071add3"} Jul 07 19:42:55.330513 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:55.330254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" event={"ID":"687df1f5-270f-4afe-afb5-472c0addf82f","Type":"ContainerStarted","Data":"8d2c9f56f760f5cb8845ea42e703b15d4da6cad94e1ae98c38950b8114801bfe"} Jul 07 19:42:55.361717 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:55.361673 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" podStartSLOduration=1.361659792 podStartE2EDuration="1.361659792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:42:55.358638855 +0000 UTC m=+2081.294264240" watchObservedRunningTime="2026-07-07 19:42:55.361659792 +0000 UTC m=+2081.297285151" Jul 07 19:42:55.688988 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:55.688911 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:55.693430 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:55.693406 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:56.336695 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:56.336667 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:56.337666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:56.337644 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:42:57.405659 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.405625 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:57.486227 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.486195 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv"] Jul 07 19:42:57.489568 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.489551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.492351 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.492329 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-778336-serving-cert\"" Jul 07 19:42:57.500185 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.500162 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv"] Jul 07 19:42:57.506861 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.506838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:57.556486 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.556455 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grlr2\" (UniqueName: \"kubernetes.io/projected/95bb7c60-25db-40ec-9196-80bafcd6d0c7-kube-api-access-grlr2\") pod \"maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.556654 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.556517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bb7c60-25db-40ec-9196-80bafcd6d0c7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.556654 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.556545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95bb7c60-25db-40ec-9196-80bafcd6d0c7-maas-api-tls\") pod \"maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.609207 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.609166 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:57.657913 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.657799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grlr2\" (UniqueName: \"kubernetes.io/projected/95bb7c60-25db-40ec-9196-80bafcd6d0c7-kube-api-access-grlr2\") pod \"maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.657913 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.657862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bb7c60-25db-40ec-9196-80bafcd6d0c7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.658155 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.657925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95bb7c60-25db-40ec-9196-80bafcd6d0c7-maas-api-tls\") pod \"maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.658652 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.658635 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bb7c60-25db-40ec-9196-80bafcd6d0c7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.660459 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.660432 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95bb7c60-25db-40ec-9196-80bafcd6d0c7-maas-api-tls\") pod \"maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.668310 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.668285 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grlr2\" (UniqueName: \"kubernetes.io/projected/95bb7c60-25db-40ec-9196-80bafcd6d0c7-kube-api-access-grlr2\") pod \"maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.707195 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.707155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:57.801964 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.801933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:57.945376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:57.945349 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv"] Jul 07 19:42:57.946928 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:42:57.946898 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bb7c60_25db_40ec_9196_80bafcd6d0c7.slice/crio-1210d871700196993a19d11add48fc64f361c4cf6b5a9077cf4fac8d077bf1ab WatchSource:0}: Error finding container 1210d871700196993a19d11add48fc64f361c4cf6b5a9077cf4fac8d077bf1ab: Status 404 returned error can't find the container with id 1210d871700196993a19d11add48fc64f361c4cf6b5a9077cf4fac8d077bf1ab Jul 07 19:42:58.007775 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:58.007745 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:58.121722 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:58.121683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:58.348363 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:58.348326 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" event={"ID":"95bb7c60-25db-40ec-9196-80bafcd6d0c7","Type":"ContainerStarted","Data":"1210d871700196993a19d11add48fc64f361c4cf6b5a9077cf4fac8d077bf1ab"} Jul 07 19:42:59.356193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:59.356156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" event={"ID":"95bb7c60-25db-40ec-9196-80bafcd6d0c7","Type":"ContainerStarted","Data":"8dd31d3fa3c270c4154d7e45fc78de2dc6491e9ae5b074ac479ea4392ff454a5"} Jul 07 19:42:59.356581 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:59.356246 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:42:59.386628 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:42:59.386573 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" podStartSLOduration=1.97325514 podStartE2EDuration="2.386554875s" totalImagesPullingTime="413.299735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:42:59.38492309 +0000 UTC m=+2085.320548638" watchObservedRunningTime="2026-07-07 19:42:59.386554875 +0000 UTC m=+2085.322180236" Jul 07 19:43:02.028043 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:02.028009 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:02.384930 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:02.384832 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:02.874710 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:02.874668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:03.067386 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:03.067350 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:05.367690 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:05.367658 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:43:06.162995 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.162911 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:06.272017 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.271987 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv"] Jul 07 19:43:06.273309 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.273283 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:06.276325 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.276306 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.281378 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.281347 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-fe4a68-serving-cert\"" Jul 07 19:43:06.292905 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.292022 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv"] Jul 07 19:43:06.345459 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.345426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3ae93d-8423-4c02-857a-fdadcaf41196-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.345621 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.345500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhgn\" (UniqueName: \"kubernetes.io/projected/6d3ae93d-8423-4c02-857a-fdadcaf41196-kube-api-access-klhgn\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.345621 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.345538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3ae93d-8423-4c02-857a-fdadcaf41196-maas-api-tls\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.370189 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.370153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:06.446934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.446849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3ae93d-8423-4c02-857a-fdadcaf41196-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.447083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.446945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhgn\" (UniqueName: \"kubernetes.io/projected/6d3ae93d-8423-4c02-857a-fdadcaf41196-kube-api-access-klhgn\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.447083 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.446967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3ae93d-8423-4c02-857a-fdadcaf41196-maas-api-tls\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.447083 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:43:06.447076 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-fe4a68-serving-cert: secret "maas-api-e2e-shared-b-fe4a68-serving-cert" not found Jul 07 19:43:06.447190 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:43:06.447135 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3ae93d-8423-4c02-857a-fdadcaf41196-maas-api-tls podName:6d3ae93d-8423-4c02-857a-fdadcaf41196 nodeName:}" failed. No retries permitted until 2026-07-07 19:43:06.94712045 +0000 UTC m=+2092.882745788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d3ae93d-8423-4c02-857a-fdadcaf41196-maas-api-tls") pod "maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" (UID: "6d3ae93d-8423-4c02-857a-fdadcaf41196") : secret "maas-api-e2e-shared-b-fe4a68-serving-cert" not found Jul 07 19:43:06.447505 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.447487 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3ae93d-8423-4c02-857a-fdadcaf41196-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.457165 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.457143 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhgn\" (UniqueName: \"kubernetes.io/projected/6d3ae93d-8423-4c02-857a-fdadcaf41196-kube-api-access-klhgn\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.551591 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.551552 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:06.865688 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.865647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:06.952630 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.952597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3ae93d-8423-4c02-857a-fdadcaf41196-maas-api-tls\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.955123 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.955097 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3ae93d-8423-4c02-857a-fdadcaf41196-maas-api-tls\") pod \"maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:06.964100 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:06.964072 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:07.217462 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:07.217357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:07.403906 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:07.395897 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv"] Jul 07 19:43:07.413628 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:43:07.413600 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3ae93d_8423_4c02_857a_fdadcaf41196.slice/crio-bd9d2a7455e0c894030af74b7d9d0bdc38e9c3e66f058b624db5a43f8a2f6ba5 WatchSource:0}: Error finding container bd9d2a7455e0c894030af74b7d9d0bdc38e9c3e66f058b624db5a43f8a2f6ba5: Status 404 returned error can't find the container with id bd9d2a7455e0c894030af74b7d9d0bdc38e9c3e66f058b624db5a43f8a2f6ba5 Jul 07 19:43:08.426339 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:08.426301 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" event={"ID":"6d3ae93d-8423-4c02-857a-fdadcaf41196","Type":"ContainerStarted","Data":"6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821"} Jul 07 19:43:08.426731 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:08.426375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" event={"ID":"6d3ae93d-8423-4c02-857a-fdadcaf41196","Type":"ContainerStarted","Data":"bd9d2a7455e0c894030af74b7d9d0bdc38e9c3e66f058b624db5a43f8a2f6ba5"} Jul 07 19:43:08.426731 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:08.426414 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:10.682841 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:10.682787 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:11.338129 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:11.338091 2577 patch_prober.go:28] interesting pod/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 07 19:43:11.338366 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:11.338150 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 07 19:43:11.447497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:11.446331 2577 generic.go:356] "Generic (PLEG): container finished" podID="687df1f5-270f-4afe-afb5-472c0addf82f" containerID="cffd0bc89acc063daef36c736ae3111b221e91d00043d314d164f79f2071add3" exitCode=0 Jul 07 19:43:11.447497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:11.446381 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" event={"ID":"687df1f5-270f-4afe-afb5-472c0addf82f","Type":"ContainerDied","Data":"cffd0bc89acc063daef36c736ae3111b221e91d00043d314d164f79f2071add3"} Jul 07 19:43:11.447497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:11.446817 2577 scope.go:117] "RemoveContainer" containerID="cffd0bc89acc063daef36c736ae3111b221e91d00043d314d164f79f2071add3" Jul 07 19:43:11.448388 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:11.448163 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:43:11.448388 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:11.448248 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:43:11.448388 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:11.448291 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 19:43:11.472896 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:11.472588 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" podStartSLOduration=5.172108269 podStartE2EDuration="5.472569735s" totalImagesPullingTime="300.461466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:43:08.447836335 +0000 UTC m=+2094.383461696" watchObservedRunningTime="2026-07-07 19:43:11.472569735 +0000 UTC m=+2097.408195096" Jul 07 19:43:12.453139 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:12.453104 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" event={"ID":"687df1f5-270f-4afe-afb5-472c0addf82f","Type":"ContainerStarted","Data":"64667311774d55c0ee77cc6ba8c7f3cd6d88f5c6ba154a60d4962ff54664b7b5"} Jul 07 19:43:12.695909 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:12.692292 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:43:12.695909 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:12.692413 2577 patch_prober.go:28] interesting pod/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 07 19:43:12.695909 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:12.692457 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 07 19:43:13.689814 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:13.689768 2577 patch_prober.go:28] interesting pod/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 07 19:43:13.690355 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:13.689837 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 07 19:43:14.438852 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:14.438825 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:43:14.689337 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:14.689240 2577 patch_prober.go:28] interesting pod/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 07 19:43:14.689624 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:14.689593 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 07 19:43:15.689643 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:15.689604 2577 patch_prober.go:28] interesting pod/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 07 19:43:15.690255 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:15.690222 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 07 19:43:16.693636 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:16.693608 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:43:16.694129 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:16.693902 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:43:16.694649 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:16.694614 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:43:20.010106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:20.010078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:20.105371 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:20.105319 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:25.273205 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:25.273175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:27.759034 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:27.759001 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:27.853613 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:27.853576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:59.963994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:43:59.963940 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:04.536308 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:44:04.536273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:04.631688 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:44:04.631658 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:54.788071 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:44:54.788025 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:00.150924 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.150865 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k"] Jul 07 19:45:00.155105 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.155083 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724225-2j7d5"] Jul 07 19:45:00.155215 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.155194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" Jul 07 19:45:00.158580 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.158558 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm"] Jul 07 19:45:00.158713 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.158661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" Jul 07 19:45:00.161679 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.161660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" Jul 07 19:45:00.164634 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.164606 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k"] Jul 07 19:45:00.169755 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.169727 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724225-2j7d5"] Jul 07 19:45:00.172501 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.172461 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm"] Jul 07 19:45:00.283644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.283610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hn6r\" (UniqueName: \"kubernetes.io/projected/2dd6e878-8025-4f8f-9a4f-acad58402755-kube-api-access-5hn6r\") pod \"maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm\" (UID: \"2dd6e878-8025-4f8f-9a4f-acad58402755\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" Jul 07 19:45:00.283644 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.283652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xxw\" (UniqueName: \"kubernetes.io/projected/d14ac054-aedd-4268-acb7-c83307b87b3b-kube-api-access-d8xxw\") pod \"maas-api-key-cleanup-29724225-2j7d5\" (UID: \"d14ac054-aedd-4268-acb7-c83307b87b3b\") " pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" Jul 07 19:45:00.283905 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.283741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/29d5aa4a-29f9-45ca-a851-2886cff4cd40-kube-api-access-bltdp\") pod \"maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k\" (UID: \"29d5aa4a-29f9-45ca-a851-2886cff4cd40\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" Jul 07 19:45:00.384718 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.384684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hn6r\" (UniqueName: \"kubernetes.io/projected/2dd6e878-8025-4f8f-9a4f-acad58402755-kube-api-access-5hn6r\") pod \"maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm\" (UID: \"2dd6e878-8025-4f8f-9a4f-acad58402755\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" Jul 07 19:45:00.384718 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.384725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xxw\" (UniqueName: \"kubernetes.io/projected/d14ac054-aedd-4268-acb7-c83307b87b3b-kube-api-access-d8xxw\") pod \"maas-api-key-cleanup-29724225-2j7d5\" (UID: \"d14ac054-aedd-4268-acb7-c83307b87b3b\") " pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" Jul 07 19:45:00.384967 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.384917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/29d5aa4a-29f9-45ca-a851-2886cff4cd40-kube-api-access-bltdp\") pod \"maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k\" (UID: \"29d5aa4a-29f9-45ca-a851-2886cff4cd40\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" Jul 07 19:45:00.395593 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.395511 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hn6r\" (UniqueName: \"kubernetes.io/projected/2dd6e878-8025-4f8f-9a4f-acad58402755-kube-api-access-5hn6r\") pod \"maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm\" (UID: \"2dd6e878-8025-4f8f-9a4f-acad58402755\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" Jul 07 19:45:00.397541 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.397518 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/29d5aa4a-29f9-45ca-a851-2886cff4cd40-kube-api-access-bltdp\") pod \"maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k\" (UID: \"29d5aa4a-29f9-45ca-a851-2886cff4cd40\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" Jul 07 19:45:00.398072 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.398051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xxw\" (UniqueName: \"kubernetes.io/projected/d14ac054-aedd-4268-acb7-c83307b87b3b-kube-api-access-d8xxw\") pod \"maas-api-key-cleanup-29724225-2j7d5\" (UID: \"d14ac054-aedd-4268-acb7-c83307b87b3b\") " pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" Jul 07 19:45:00.469267 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.469186 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" Jul 07 19:45:00.481394 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.481361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" Jul 07 19:45:00.488315 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.488282 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" Jul 07 19:45:00.644488 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.644466 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k"] Jul 07 19:45:00.646484 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:45:00.646457 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d5aa4a_29f9_45ca_a851_2886cff4cd40.slice/crio-f6359fe609b930f0283ea6390075f725757526f62d06c44b9f336cc3062d909c WatchSource:0}: Error finding container f6359fe609b930f0283ea6390075f725757526f62d06c44b9f336cc3062d909c: Status 404 returned error can't find the container with id f6359fe609b930f0283ea6390075f725757526f62d06c44b9f336cc3062d909c Jul 07 19:45:00.648335 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.648319 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:45:00.879146 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.879044 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm"] Jul 07 19:45:00.880793 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:45:00.880762 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd6e878_8025_4f8f_9a4f_acad58402755.slice/crio-08ad2036b43643979abede77895a24c5f2984190e946ba1f76b0d0cfcb1e8f59 WatchSource:0}: Error finding container 08ad2036b43643979abede77895a24c5f2984190e946ba1f76b0d0cfcb1e8f59: Status 404 returned error can't find the container with id 08ad2036b43643979abede77895a24c5f2984190e946ba1f76b0d0cfcb1e8f59 Jul 07 19:45:00.881944 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:00.881917 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724225-2j7d5"] Jul 07 19:45:00.883707 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:45:00.883677 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14ac054_aedd_4268_acb7_c83307b87b3b.slice/crio-d1c191e9e4050cf9e88e0681bb636f23e3f8aa7459be468a11e0510d97a48da3 WatchSource:0}: Error finding container d1c191e9e4050cf9e88e0681bb636f23e3f8aa7459be468a11e0510d97a48da3: Status 404 returned error can't find the container with id d1c191e9e4050cf9e88e0681bb636f23e3f8aa7459be468a11e0510d97a48da3 Jul 07 19:45:01.021371 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:01.021337 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" event={"ID":"29d5aa4a-29f9-45ca-a851-2886cff4cd40","Type":"ContainerStarted","Data":"f6359fe609b930f0283ea6390075f725757526f62d06c44b9f336cc3062d909c"} Jul 07 19:45:01.022511 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:01.022483 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" event={"ID":"2dd6e878-8025-4f8f-9a4f-acad58402755","Type":"ContainerStarted","Data":"08ad2036b43643979abede77895a24c5f2984190e946ba1f76b0d0cfcb1e8f59"} Jul 07 19:45:01.023497 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:01.023474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" event={"ID":"d14ac054-aedd-4268-acb7-c83307b87b3b","Type":"ContainerStarted","Data":"d1c191e9e4050cf9e88e0681bb636f23e3f8aa7459be468a11e0510d97a48da3"} Jul 07 19:45:04.045289 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:04.045248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" event={"ID":"d14ac054-aedd-4268-acb7-c83307b87b3b","Type":"ContainerStarted","Data":"fe4a2b8d6eaddd95dc43da97873a80e83bab63677b12f38d4afb5b56d242290e"} Jul 07 19:45:04.046797 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:04.046680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" event={"ID":"29d5aa4a-29f9-45ca-a851-2886cff4cd40","Type":"ContainerStarted","Data":"adb20812004951e29742719f5d63ca8f1283de85209e66ae5d84c040afa6f7ad"} Jul 07 19:45:04.048161 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:04.048140 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" event={"ID":"2dd6e878-8025-4f8f-9a4f-acad58402755","Type":"ContainerStarted","Data":"51552c2abcc668d0f82c138d06473c923cc5caa3e20675808cc74288092d34cb"} Jul 07 19:45:04.069389 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:04.069345 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" podStartSLOduration=1.639258935 podStartE2EDuration="4.069321885s" totalImagesPullingTime="2.43006295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:45:04.0690406 +0000 UTC m=+2210.004665961" watchObservedRunningTime="2026-07-07 19:45:04.069321885 +0000 UTC m=+2210.004947244" Jul 07 19:45:04.092300 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:04.092240 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" podStartSLOduration=1.424013835 podStartE2EDuration="4.092225063s" totalImagesPullingTime="2.668211228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:45:04.090070204 +0000 UTC m=+2210.025695591" watchObservedRunningTime="2026-07-07 19:45:04.092225063 +0000 UTC m=+2210.027850424" Jul 07 19:45:04.110899 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:04.110829 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" podStartSLOduration=1.7794749479999998 podStartE2EDuration="4.1108141s" totalImagesPullingTime="2.331339152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:45:04.108520648 +0000 UTC m=+2210.044146020" watchObservedRunningTime="2026-07-07 19:45:04.1108141 +0000 UTC m=+2210.046439460" Jul 07 19:45:05.054889 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:05.054851 2577 generic.go:356] "Generic (PLEG): container finished" podID="d14ac054-aedd-4268-acb7-c83307b87b3b" containerID="fe4a2b8d6eaddd95dc43da97873a80e83bab63677b12f38d4afb5b56d242290e" exitCode=0 Jul 07 19:45:05.055292 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:05.054928 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" event={"ID":"d14ac054-aedd-4268-acb7-c83307b87b3b","Type":"ContainerDied","Data":"fe4a2b8d6eaddd95dc43da97873a80e83bab63677b12f38d4afb5b56d242290e"} Jul 07 19:45:05.056383 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:05.056360 2577 generic.go:356] "Generic (PLEG): container finished" podID="29d5aa4a-29f9-45ca-a851-2886cff4cd40" containerID="adb20812004951e29742719f5d63ca8f1283de85209e66ae5d84c040afa6f7ad" exitCode=0 Jul 07 19:45:05.056498 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:05.056388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" event={"ID":"29d5aa4a-29f9-45ca-a851-2886cff4cd40","Type":"ContainerDied","Data":"adb20812004951e29742719f5d63ca8f1283de85209e66ae5d84c040afa6f7ad"} Jul 07 19:45:05.057778 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:05.057757 2577 generic.go:356] "Generic (PLEG): container finished" podID="2dd6e878-8025-4f8f-9a4f-acad58402755" containerID="51552c2abcc668d0f82c138d06473c923cc5caa3e20675808cc74288092d34cb" exitCode=0 Jul 07 19:45:05.057859 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:05.057787 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" event={"ID":"2dd6e878-8025-4f8f-9a4f-acad58402755","Type":"ContainerDied","Data":"51552c2abcc668d0f82c138d06473c923cc5caa3e20675808cc74288092d34cb"} Jul 07 19:45:06.222064 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.222034 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" Jul 07 19:45:06.263611 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.263585 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" Jul 07 19:45:06.267390 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.267326 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" Jul 07 19:45:06.343113 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.343040 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8xxw\" (UniqueName: \"kubernetes.io/projected/d14ac054-aedd-4268-acb7-c83307b87b3b-kube-api-access-d8xxw\") pod \"d14ac054-aedd-4268-acb7-c83307b87b3b\" (UID: \"d14ac054-aedd-4268-acb7-c83307b87b3b\") " Jul 07 19:45:06.343113 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.343082 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/29d5aa4a-29f9-45ca-a851-2886cff4cd40-kube-api-access-bltdp\") pod \"29d5aa4a-29f9-45ca-a851-2886cff4cd40\" (UID: \"29d5aa4a-29f9-45ca-a851-2886cff4cd40\") " Jul 07 19:45:06.343113 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.343103 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hn6r\" (UniqueName: \"kubernetes.io/projected/2dd6e878-8025-4f8f-9a4f-acad58402755-kube-api-access-5hn6r\") pod \"2dd6e878-8025-4f8f-9a4f-acad58402755\" (UID: \"2dd6e878-8025-4f8f-9a4f-acad58402755\") " Jul 07 19:45:06.345190 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.345159 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d5aa4a-29f9-45ca-a851-2886cff4cd40-kube-api-access-bltdp" (OuterVolumeSpecName: "kube-api-access-bltdp") pod "29d5aa4a-29f9-45ca-a851-2886cff4cd40" (UID: "29d5aa4a-29f9-45ca-a851-2886cff4cd40"). InnerVolumeSpecName "kube-api-access-bltdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:45:06.345351 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.345300 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14ac054-aedd-4268-acb7-c83307b87b3b-kube-api-access-d8xxw" (OuterVolumeSpecName: "kube-api-access-d8xxw") pod "d14ac054-aedd-4268-acb7-c83307b87b3b" (UID: "d14ac054-aedd-4268-acb7-c83307b87b3b"). InnerVolumeSpecName "kube-api-access-d8xxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:45:06.345351 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.345335 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd6e878-8025-4f8f-9a4f-acad58402755-kube-api-access-5hn6r" (OuterVolumeSpecName: "kube-api-access-5hn6r") pod "2dd6e878-8025-4f8f-9a4f-acad58402755" (UID: "2dd6e878-8025-4f8f-9a4f-acad58402755"). InnerVolumeSpecName "kube-api-access-5hn6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:45:06.444594 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.444554 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8xxw\" (UniqueName: \"kubernetes.io/projected/d14ac054-aedd-4268-acb7-c83307b87b3b-kube-api-access-d8xxw\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:45:06.444594 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.444584 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/29d5aa4a-29f9-45ca-a851-2886cff4cd40-kube-api-access-bltdp\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:45:06.444594 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:06.444595 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hn6r\" (UniqueName: \"kubernetes.io/projected/2dd6e878-8025-4f8f-9a4f-acad58402755-kube-api-access-5hn6r\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:45:07.069522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:07.069487 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" Jul 07 19:45:07.069691 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:07.069518 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm" event={"ID":"2dd6e878-8025-4f8f-9a4f-acad58402755","Type":"ContainerDied","Data":"08ad2036b43643979abede77895a24c5f2984190e946ba1f76b0d0cfcb1e8f59"} Jul 07 19:45:07.069691 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:07.069572 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08ad2036b43643979abede77895a24c5f2984190e946ba1f76b0d0cfcb1e8f59" Jul 07 19:45:07.070821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:07.070802 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" Jul 07 19:45:07.070994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:07.070843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724225-2j7d5" event={"ID":"d14ac054-aedd-4268-acb7-c83307b87b3b","Type":"ContainerDied","Data":"d1c191e9e4050cf9e88e0681bb636f23e3f8aa7459be468a11e0510d97a48da3"} Jul 07 19:45:07.070994 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:07.070900 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1c191e9e4050cf9e88e0681bb636f23e3f8aa7459be468a11e0510d97a48da3" Jul 07 19:45:07.072385 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:07.072370 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" Jul 07 19:45:07.072483 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:07.072366 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k" event={"ID":"29d5aa4a-29f9-45ca-a851-2886cff4cd40","Type":"ContainerDied","Data":"f6359fe609b930f0283ea6390075f725757526f62d06c44b9f336cc3062d909c"} Jul 07 19:45:07.072529 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:45:07.072490 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6359fe609b930f0283ea6390075f725757526f62d06c44b9f336cc3062d909c" Jul 07 19:46:06.716707 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:46:06.716619 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:46:08.010649 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:46:08.010614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:46:19.808580 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:46:19.808546 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:47:01.611240 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:47:01.611200 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:47:07.300995 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:47:07.300902 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:47:07.404955 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:47:07.404924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:47:28.140268 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:47:28.140232 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:47:39.308243 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:47:39.308204 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:47:52.756292 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:47:52.756248 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:47:54.671841 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:47:54.671800 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:47:55.579861 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:47:55.579806 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:04.711830 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:48:04.711792 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:20.929899 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:48:20.929817 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:21.042744 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:48:21.042694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:23.323034 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:48:23.322995 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:28.674934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:48:28.674901 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:28.776394 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:48:28.776362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:56.269616 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:48:56.269522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:14.296236 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:14.296202 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:15.130184 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:15.130144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:34.349513 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:34.349475 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:34.771720 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:34.771665 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:35.011399 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.011363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:35.116009 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.115925 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:35.295240 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.295210 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv"] Jul 07 19:49:35.295417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.295397 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" podUID="95bb7c60-25db-40ec-9196-80bafcd6d0c7" containerName="maas-api" containerID="cri-o://8dd31d3fa3c270c4154d7e45fc78de2dc6491e9ae5b074ac479ea4392ff454a5" gracePeriod=30 Jul 07 19:49:35.355049 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.355020 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm"] Jul 07 19:49:35.357057 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.357026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:35.358735 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.358711 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-778336-29724225-lwwhm"] Jul 07 19:49:35.361587 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.361557 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" podUID="95bb7c60-25db-40ec-9196-80bafcd6d0c7" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.60:8443/health\": dial tcp 10.134.0.60:8443: connect: connection refused" Jul 07 19:49:35.535802 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.535778 2577 generic.go:356] "Generic (PLEG): container finished" podID="95bb7c60-25db-40ec-9196-80bafcd6d0c7" containerID="8dd31d3fa3c270c4154d7e45fc78de2dc6491e9ae5b074ac479ea4392ff454a5" exitCode=0 Jul 07 19:49:35.535971 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.535846 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" event={"ID":"95bb7c60-25db-40ec-9196-80bafcd6d0c7","Type":"ContainerDied","Data":"8dd31d3fa3c270c4154d7e45fc78de2dc6491e9ae5b074ac479ea4392ff454a5"} Jul 07 19:49:35.553059 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.553039 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:49:35.728796 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.728706 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grlr2\" (UniqueName: \"kubernetes.io/projected/95bb7c60-25db-40ec-9196-80bafcd6d0c7-kube-api-access-grlr2\") pod \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " Jul 07 19:49:35.728987 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.728828 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bb7c60-25db-40ec-9196-80bafcd6d0c7-openshift-service-ca\") pod \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " Jul 07 19:49:35.728987 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.728923 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95bb7c60-25db-40ec-9196-80bafcd6d0c7-maas-api-tls\") pod \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\" (UID: \"95bb7c60-25db-40ec-9196-80bafcd6d0c7\") " Jul 07 19:49:35.729171 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.729139 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95bb7c60-25db-40ec-9196-80bafcd6d0c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95bb7c60-25db-40ec-9196-80bafcd6d0c7" (UID: "95bb7c60-25db-40ec-9196-80bafcd6d0c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:49:35.729326 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.729303 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bb7c60-25db-40ec-9196-80bafcd6d0c7-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:35.731067 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.731042 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95bb7c60-25db-40ec-9196-80bafcd6d0c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95bb7c60-25db-40ec-9196-80bafcd6d0c7" (UID: "95bb7c60-25db-40ec-9196-80bafcd6d0c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:49:35.731173 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.731122 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bb7c60-25db-40ec-9196-80bafcd6d0c7-kube-api-access-grlr2" (OuterVolumeSpecName: "kube-api-access-grlr2") pod "95bb7c60-25db-40ec-9196-80bafcd6d0c7" (UID: "95bb7c60-25db-40ec-9196-80bafcd6d0c7"). InnerVolumeSpecName "kube-api-access-grlr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:49:35.830779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.830745 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95bb7c60-25db-40ec-9196-80bafcd6d0c7-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:35.830779 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:35.830777 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grlr2\" (UniqueName: \"kubernetes.io/projected/95bb7c60-25db-40ec-9196-80bafcd6d0c7-kube-api-access-grlr2\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:36.543043 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:36.543018 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" Jul 07 19:49:36.543043 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:36.543031 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv" event={"ID":"95bb7c60-25db-40ec-9196-80bafcd6d0c7","Type":"ContainerDied","Data":"1210d871700196993a19d11add48fc64f361c4cf6b5a9077cf4fac8d077bf1ab"} Jul 07 19:49:36.543537 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:36.543085 2577 scope.go:117] "RemoveContainer" containerID="8dd31d3fa3c270c4154d7e45fc78de2dc6491e9ae5b074ac479ea4392ff454a5" Jul 07 19:49:36.567437 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:36.567408 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv"] Jul 07 19:49:36.572773 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:36.572754 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-778336-7dcd546c6c-jf4cv"] Jul 07 19:49:36.612265 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:36.612239 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd6e878-8025-4f8f-9a4f-acad58402755" path="/var/lib/kubelet/pods/2dd6e878-8025-4f8f-9a4f-acad58402755/volumes" Jul 07 19:49:36.612640 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:36.612626 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95bb7c60-25db-40ec-9196-80bafcd6d0c7" path="/var/lib/kubelet/pods/95bb7c60-25db-40ec-9196-80bafcd6d0c7/volumes" Jul 07 19:49:42.109940 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:42.109908 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:42.189818 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:42.189784 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:42.384995 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:42.384912 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5"] Jul 07 19:49:42.385142 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:42.385119 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" containerID="cri-o://64667311774d55c0ee77cc6ba8c7f3cd6d88f5c6ba154a60d4962ff54664b7b5" gracePeriod=30 Jul 07 19:49:42.404780 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:42.404749 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:42.630522 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:42.630479 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:42.852072 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:42.852038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:43.234343 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:43.234246 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:43.482022 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:43.481987 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:43.635589 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:43.635558 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:43.979269 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:43.979188 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv"] Jul 07 19:49:43.979466 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:43.979441 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" podUID="6d3ae93d-8423-4c02-857a-fdadcaf41196" containerName="maas-api" containerID="cri-o://6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821" gracePeriod=30 Jul 07 19:49:44.012912 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.012857 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:44.044767 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.044744 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k"] Jul 07 19:49:44.048097 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.048078 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-fe4a68-29724225-gw64k"] Jul 07 19:49:44.228620 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.228597 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:49:44.306689 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.306655 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3ae93d-8423-4c02-857a-fdadcaf41196-maas-api-tls\") pod \"6d3ae93d-8423-4c02-857a-fdadcaf41196\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " Jul 07 19:49:44.307114 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.306767 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3ae93d-8423-4c02-857a-fdadcaf41196-openshift-service-ca\") pod \"6d3ae93d-8423-4c02-857a-fdadcaf41196\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " Jul 07 19:49:44.307114 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.306790 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klhgn\" (UniqueName: \"kubernetes.io/projected/6d3ae93d-8423-4c02-857a-fdadcaf41196-kube-api-access-klhgn\") pod \"6d3ae93d-8423-4c02-857a-fdadcaf41196\" (UID: \"6d3ae93d-8423-4c02-857a-fdadcaf41196\") " Jul 07 19:49:44.307200 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.307159 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3ae93d-8423-4c02-857a-fdadcaf41196-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d3ae93d-8423-4c02-857a-fdadcaf41196" (UID: "6d3ae93d-8423-4c02-857a-fdadcaf41196"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:49:44.309189 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.309165 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3ae93d-8423-4c02-857a-fdadcaf41196-kube-api-access-klhgn" (OuterVolumeSpecName: "kube-api-access-klhgn") pod "6d3ae93d-8423-4c02-857a-fdadcaf41196" (UID: "6d3ae93d-8423-4c02-857a-fdadcaf41196"). InnerVolumeSpecName "kube-api-access-klhgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:49:44.309325 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.309224 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3ae93d-8423-4c02-857a-fdadcaf41196-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d3ae93d-8423-4c02-857a-fdadcaf41196" (UID: "6d3ae93d-8423-4c02-857a-fdadcaf41196"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:49:44.408177 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.408145 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3ae93d-8423-4c02-857a-fdadcaf41196-openshift-service-ca\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:44.408177 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.408172 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klhgn\" (UniqueName: \"kubernetes.io/projected/6d3ae93d-8423-4c02-857a-fdadcaf41196-kube-api-access-klhgn\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:44.408177 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.408181 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3ae93d-8423-4c02-857a-fdadcaf41196-maas-api-tls\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:44.590116 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.590034 2577 generic.go:356] "Generic (PLEG): container finished" podID="6d3ae93d-8423-4c02-857a-fdadcaf41196" containerID="6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821" exitCode=0 Jul 07 19:49:44.590116 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.590092 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" Jul 07 19:49:44.590291 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.590125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" event={"ID":"6d3ae93d-8423-4c02-857a-fdadcaf41196","Type":"ContainerDied","Data":"6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821"} Jul 07 19:49:44.590291 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.590184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv" event={"ID":"6d3ae93d-8423-4c02-857a-fdadcaf41196","Type":"ContainerDied","Data":"bd9d2a7455e0c894030af74b7d9d0bdc38e9c3e66f058b624db5a43f8a2f6ba5"} Jul 07 19:49:44.590291 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.590208 2577 scope.go:117] "RemoveContainer" containerID="6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821" Jul 07 19:49:44.601106 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.601092 2577 scope.go:117] "RemoveContainer" containerID="6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821" Jul 07 19:49:44.601357 ip-10-0-132-59 kubenswrapper[2577]: E0707 19:49:44.601339 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821\": container with ID starting with 6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821 not found: ID does not exist" containerID="6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821" Jul 07 19:49:44.601408 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.601366 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821"} err="failed to get container status \"6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821\": rpc error: code = NotFound desc = could not find container \"6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821\": container with ID starting with 6b549964ad16c0171482b284f43c40bde6438514531503f862ed1ae7d45af821 not found: ID does not exist" Jul 07 19:49:44.604934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.604914 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29d5aa4a-29f9-45ca-a851-2886cff4cd40" path="/var/lib/kubelet/pods/29d5aa4a-29f9-45ca-a851-2886cff4cd40/volumes" Jul 07 19:49:44.622654 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.622630 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv"] Jul 07 19:49:44.635330 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:44.635310 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-fe4a68-548b84574d-jm2zv"] Jul 07 19:49:46.604053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:46.604021 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3ae93d-8423-4c02-857a-fdadcaf41196" path="/var/lib/kubelet/pods/6d3ae93d-8423-4c02-857a-fdadcaf41196/volumes" Jul 07 19:49:46.696912 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:46.696862 2577 patch_prober.go:28] interesting pod/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 19:49:46.697068 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:46.696936 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:49:47.608807 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.608776 2577 generic.go:356] "Generic (PLEG): container finished" podID="687df1f5-270f-4afe-afb5-472c0addf82f" containerID="64667311774d55c0ee77cc6ba8c7f3cd6d88f5c6ba154a60d4962ff54664b7b5" exitCode=0 Jul 07 19:49:47.609188 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.608854 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" event={"ID":"687df1f5-270f-4afe-afb5-472c0addf82f","Type":"ContainerDied","Data":"64667311774d55c0ee77cc6ba8c7f3cd6d88f5c6ba154a60d4962ff54664b7b5"} Jul 07 19:49:47.609188 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.608936 2577 scope.go:117] "RemoveContainer" containerID="cffd0bc89acc063daef36c736ae3111b221e91d00043d314d164f79f2071add3" Jul 07 19:49:47.650274 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.650219 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:49:47.738524 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738492 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-istio-token\") pod \"687df1f5-270f-4afe-afb5-472c0addf82f\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " Jul 07 19:49:47.738524 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738530 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbz67\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-kube-api-access-vbz67\") pod \"687df1f5-270f-4afe-afb5-472c0addf82f\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " Jul 07 19:49:47.738771 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738558 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-socket\") pod \"687df1f5-270f-4afe-afb5-472c0addf82f\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " Jul 07 19:49:47.738771 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738595 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-certs\") pod \"687df1f5-270f-4afe-afb5-472c0addf82f\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " Jul 07 19:49:47.738771 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738648 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687df1f5-270f-4afe-afb5-472c0addf82f-istiod-ca-cert\") pod \"687df1f5-270f-4afe-afb5-472c0addf82f\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " Jul 07 19:49:47.738771 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-data\") pod \"687df1f5-270f-4afe-afb5-472c0addf82f\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " Jul 07 19:49:47.738771 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738756 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687df1f5-270f-4afe-afb5-472c0addf82f-istio-podinfo\") pod \"687df1f5-270f-4afe-afb5-472c0addf82f\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " Jul 07 19:49:47.739089 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738787 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-envoy\") pod \"687df1f5-270f-4afe-afb5-472c0addf82f\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " Jul 07 19:49:47.739089 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738824 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-credential-socket\") pod \"687df1f5-270f-4afe-afb5-472c0addf82f\" (UID: \"687df1f5-270f-4afe-afb5-472c0addf82f\") " Jul 07 19:49:47.739089 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.738961 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "687df1f5-270f-4afe-afb5-472c0addf82f" (UID: "687df1f5-270f-4afe-afb5-472c0addf82f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:49:47.739254 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.739095 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "687df1f5-270f-4afe-afb5-472c0addf82f" (UID: "687df1f5-270f-4afe-afb5-472c0addf82f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:49:47.739402 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.739295 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-credential-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:47.739402 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.739318 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-socket\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:47.739583 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.739509 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687df1f5-270f-4afe-afb5-472c0addf82f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "687df1f5-270f-4afe-afb5-472c0addf82f" (UID: "687df1f5-270f-4afe-afb5-472c0addf82f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:49:47.739686 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.739667 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "687df1f5-270f-4afe-afb5-472c0addf82f" (UID: "687df1f5-270f-4afe-afb5-472c0addf82f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:49:47.739896 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.739861 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-data" (OuterVolumeSpecName: "istio-data") pod "687df1f5-270f-4afe-afb5-472c0addf82f" (UID: "687df1f5-270f-4afe-afb5-472c0addf82f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:49:47.741016 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.740995 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-kube-api-access-vbz67" (OuterVolumeSpecName: "kube-api-access-vbz67") pod "687df1f5-270f-4afe-afb5-472c0addf82f" (UID: "687df1f5-270f-4afe-afb5-472c0addf82f"). InnerVolumeSpecName "kube-api-access-vbz67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:49:47.741316 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.741280 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/687df1f5-270f-4afe-afb5-472c0addf82f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "687df1f5-270f-4afe-afb5-472c0addf82f" (UID: "687df1f5-270f-4afe-afb5-472c0addf82f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:49:47.741433 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.741395 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-istio-token" (OuterVolumeSpecName: "istio-token") pod "687df1f5-270f-4afe-afb5-472c0addf82f" (UID: "687df1f5-270f-4afe-afb5-472c0addf82f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:49:47.741505 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.741486 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "687df1f5-270f-4afe-afb5-472c0addf82f" (UID: "687df1f5-270f-4afe-afb5-472c0addf82f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:49:47.840602 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.840570 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687df1f5-270f-4afe-afb5-472c0addf82f-istio-podinfo\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:47.840602 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.840597 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-envoy\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:47.840602 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.840606 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-istio-token\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:47.840821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.840613 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbz67\" (UniqueName: \"kubernetes.io/projected/687df1f5-270f-4afe-afb5-472c0addf82f-kube-api-access-vbz67\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:47.840821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.840622 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-workload-certs\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:47.840821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.840631 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687df1f5-270f-4afe-afb5-472c0addf82f-istiod-ca-cert\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:47.840821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:47.840638 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687df1f5-270f-4afe-afb5-472c0addf82f-istio-data\") on node \"ip-10-0-132-59.ec2.internal\" DevicePath \"\"" Jul 07 19:49:48.619821 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:48.619793 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" Jul 07 19:49:48.620156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:48.619812 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5" event={"ID":"687df1f5-270f-4afe-afb5-472c0addf82f","Type":"ContainerDied","Data":"8d2c9f56f760f5cb8845ea42e703b15d4da6cad94e1ae98c38950b8114801bfe"} Jul 07 19:49:48.620156 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:48.619921 2577 scope.go:117] "RemoveContainer" containerID="64667311774d55c0ee77cc6ba8c7f3cd6d88f5c6ba154a60d4962ff54664b7b5" Jul 07 19:49:48.648540 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:48.648512 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5"] Jul 07 19:49:48.657193 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:48.657171 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-778336-openshift-default-f645995dc-xlxc5"] Jul 07 19:49:50.604188 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:50.604158 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" path="/var/lib/kubelet/pods/687df1f5-270f-4afe-afb5-472c0addf82f/volumes" Jul 07 19:49:50.785523 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:50.785493 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:50.939264 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:50.939163 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:51.236061 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:51.235971 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:51.353100 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:51.353069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:56.442389 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:56.442356 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-67ccf_468aba6c-4436-470a-a5de-34c2e24325b0/manager/0.log" Jul 07 19:49:56.559341 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:56.559313 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tt9z9_d921ac91-a173-4000-9da8-26c6037abeaa/manager/0.log" Jul 07 19:49:56.684198 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:56.684169 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6d8c95d7dc-4wkl6_f3e3e2ee-1ea8-4687-a389-bcc6e10515b4/maas-api/0.log" Jul 07 19:49:56.913999 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:56.913974 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724225-2j7d5_d14ac054-aedd-4268-acb7-c83307b87b3b/cleanup/0.log" Jul 07 19:49:57.037008 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:57.036976 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9c77bd8b4-lpqf7_acbc8b6a-958b-483e-b393-1d6b17ed660f/manager/0.log" Jul 07 19:49:57.142609 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:57.142581 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:57.238816 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:57.238719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:57.396380 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:57.396352 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8689849645-cjrtk_20d98418-bc55-46ad-a0a2-934ac44de34d/copy-manifests/0.log" Jul 07 19:49:57.403801 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:57.403778 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8689849645-cjrtk_20d98418-bc55-46ad-a0a2-934ac44de34d/manager/0.log" Jul 07 19:49:57.800382 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:57.800354 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-2bhbv_3adf1e3e-26fb-465c-b13c-cf417fc4461e/postgres/0.log" Jul 07 19:49:59.510525 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:59.510498 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-z6vwk_32837b00-0905-4f99-a9f3-8f2f61438693/manager/0.log" Jul 07 19:49:59.886272 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:49:59.886188 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vmgx5_0341e53b-1ea6-4ffa-a235-64bc0c097091/manager/0.log" Jul 07 19:50:00.530989 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:00.530957 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4wh7sr_f44ed616-49f7-49ef-8db4-ac4135c3379b/istio-proxy/0.log" Jul 07 19:50:01.058461 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:01.058432 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6nj7_93df3e10-d156-40e0-8f81-367099052b5f/istio-proxy/1.log" Jul 07 19:50:01.197545 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:01.197514 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-chkp7_3c2fb729-32ba-4b76-acb7-ab6c74fe1074/payload-pre-processing/0.log" Jul 07 19:50:01.883062 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:01.883034 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-hszjd_c00fc2de-eb4b-45de-a766-14cc32b41de6/main/0.log" Jul 07 19:50:02.003024 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:02.002994 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fbcxd_c968b2e1-f195-4583-b42e-3771722f8b5c/main/0.log" Jul 07 19:50:07.080427 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:07.080333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:50:07.159533 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:07.159500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:50:10.236661 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:10.236629 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5m825_80d3ae4b-a4d1-4f35-8815-61d9a05251ba/global-pull-secret-syncer/0.log" Jul 07 19:50:10.456479 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:10.456450 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" Jul 07 19:50:10.671436 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:10.671408 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-59.ec2.internal_36efaeb11e6e863bb0fd6bda41d8081a/haproxy/0.log" Jul 07 19:50:11.619670 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:11.619639 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-87p9q_2f4a402b-2e9f-4b07-9b1c-3b77776a6358/konnectivity-agent/0.log" limit=1024 Jul 07 19:50:16.003781 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:16.003696 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-z6vwk_32837b00-0905-4f99-a9f3-8f2f61438693/manager/0.log" Jul 07 19:50:16.341981 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:16.341895 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vmgx5_0341e53b-1ea6-4ffa-a235-64bc0c097091/manager/0.log" Jul 07 19:50:18.152807 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.152718 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d4b3b5-a308-4726-a62c-e5c2f891e671/alertmanager/0.log" Jul 07 19:50:18.184417 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.184385 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d4b3b5-a308-4726-a62c-e5c2f891e671/config-reloader/0.log" Jul 07 19:50:18.211337 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.211314 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d4b3b5-a308-4726-a62c-e5c2f891e671/kube-rbac-proxy-web/0.log" Jul 07 19:50:18.241683 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.241663 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d4b3b5-a308-4726-a62c-e5c2f891e671/kube-rbac-proxy/0.log" Jul 07 19:50:18.275299 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.275275 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d4b3b5-a308-4726-a62c-e5c2f891e671/kube-rbac-proxy-metric/0.log" Jul 07 19:50:18.308046 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.308028 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d4b3b5-a308-4726-a62c-e5c2f891e671/prom-label-proxy/0.log" Jul 07 19:50:18.347017 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.346997 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d4b3b5-a308-4726-a62c-e5c2f891e671/init-config-reloader/0.log" Jul 07 19:50:18.458451 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.458393 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-mk2p6_69501978-25c4-4e80-bb45-009fdc2e3a84/kube-state-metrics/0.log" Jul 07 19:50:18.480376 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.480328 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-mk2p6_69501978-25c4-4e80-bb45-009fdc2e3a84/kube-rbac-proxy-main/0.log" Jul 07 19:50:18.509792 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.509767 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-mk2p6_69501978-25c4-4e80-bb45-009fdc2e3a84/kube-rbac-proxy-self/0.log" Jul 07 19:50:18.544730 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.544706 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-698d7884fb-pvnc4_f8a4c4c0-2376-4a6a-9eb5-dd01da1bf2df/metrics-server/0.log" Jul 07 19:50:18.582460 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.582439 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-jt8j8_0c37c8da-3a3a-4fa0-86b3-6c6d92807b40/monitoring-plugin/0.log" Jul 07 19:50:18.830495 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.830473 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xwngj_703f6de1-91b9-4fd0-8f74-c617e8968f09/node-exporter/0.log" Jul 07 19:50:18.859513 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.859484 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xwngj_703f6de1-91b9-4fd0-8f74-c617e8968f09/kube-rbac-proxy/0.log" Jul 07 19:50:18.884666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.884646 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xwngj_703f6de1-91b9-4fd0-8f74-c617e8968f09/init-textfile/0.log" Jul 07 19:50:18.927804 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.927785 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kzplg_830d4717-38fd-4379-ac29-bb324132289b/kube-rbac-proxy-main/0.log" Jul 07 19:50:18.954385 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.954360 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kzplg_830d4717-38fd-4379-ac29-bb324132289b/kube-rbac-proxy-self/0.log" Jul 07 19:50:18.984483 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:18.984463 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kzplg_830d4717-38fd-4379-ac29-bb324132289b/openshift-state-metrics/0.log" Jul 07 19:50:19.032108 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.032086 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e88c4bd6-8796-4ba1-8527-ad3677782c0f/prometheus/0.log" Jul 07 19:50:19.070231 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.070209 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e88c4bd6-8796-4ba1-8527-ad3677782c0f/config-reloader/0.log" Jul 07 19:50:19.103988 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.103908 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e88c4bd6-8796-4ba1-8527-ad3677782c0f/thanos-sidecar/0.log" Jul 07 19:50:19.141475 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.141455 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e88c4bd6-8796-4ba1-8527-ad3677782c0f/kube-rbac-proxy-web/0.log" Jul 07 19:50:19.176465 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.176444 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e88c4bd6-8796-4ba1-8527-ad3677782c0f/kube-rbac-proxy/0.log" Jul 07 19:50:19.209488 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.209459 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e88c4bd6-8796-4ba1-8527-ad3677782c0f/kube-rbac-proxy-thanos/0.log" Jul 07 19:50:19.246148 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.246125 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e88c4bd6-8796-4ba1-8527-ad3677782c0f/init-config-reloader/0.log" Jul 07 19:50:19.287484 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.287467 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-vnsh6_1119552f-394b-40bb-b13f-8c3235e908f1/prometheus-operator/0.log" Jul 07 19:50:19.316321 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.316296 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-vnsh6_1119552f-394b-40bb-b13f-8c3235e908f1/kube-rbac-proxy/0.log" Jul 07 19:50:19.410537 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.410460 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bbdd4f87-jfknr_2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b/telemeter-client/0.log" Jul 07 19:50:19.444408 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.444386 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bbdd4f87-jfknr_2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b/reload/0.log" Jul 07 19:50:19.482117 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.482099 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bbdd4f87-jfknr_2d0a1d1e-f902-47bf-8b9a-ff2ccf1ce88b/kube-rbac-proxy/0.log" Jul 07 19:50:19.522371 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.522352 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcbcfdd9b-n26x8_ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe/thanos-query/0.log" Jul 07 19:50:19.562666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.562646 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcbcfdd9b-n26x8_ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe/kube-rbac-proxy-web/0.log" Jul 07 19:50:19.591402 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.591381 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcbcfdd9b-n26x8_ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe/kube-rbac-proxy/0.log" Jul 07 19:50:19.619370 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.619354 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcbcfdd9b-n26x8_ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe/prom-label-proxy/0.log" Jul 07 19:50:19.646002 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.645970 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcbcfdd9b-n26x8_ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe/kube-rbac-proxy-rules/0.log" Jul 07 19:50:19.677071 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.676995 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcbcfdd9b-n26x8_ba312e2d-2c39-45d6-bcd6-e9017a1c9ffe/kube-rbac-proxy-metrics/0.log" Jul 07 19:50:19.967132 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967054 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r"] Jul 07 19:50:19.967478 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967467 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95bb7c60-25db-40ec-9196-80bafcd6d0c7" containerName="maas-api" Jul 07 19:50:19.967521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967480 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95bb7c60-25db-40ec-9196-80bafcd6d0c7" containerName="maas-api" Jul 07 19:50:19.967521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967488 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" Jul 07 19:50:19.967521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967493 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" Jul 07 19:50:19.967521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967501 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29d5aa4a-29f9-45ca-a851-2886cff4cd40" containerName="cleanup" Jul 07 19:50:19.967521 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967507 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29d5aa4a-29f9-45ca-a851-2886cff4cd40" containerName="cleanup" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967527 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3ae93d-8423-4c02-857a-fdadcaf41196" containerName="maas-api" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967532 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3ae93d-8423-4c02-857a-fdadcaf41196" containerName="maas-api" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967538 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd6e878-8025-4f8f-9a4f-acad58402755" containerName="cleanup" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967544 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd6e878-8025-4f8f-9a4f-acad58402755" containerName="cleanup" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967556 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d14ac054-aedd-4268-acb7-c83307b87b3b" containerName="cleanup" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967560 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d14ac054-aedd-4268-acb7-c83307b87b3b" containerName="cleanup" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967620 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95bb7c60-25db-40ec-9196-80bafcd6d0c7" containerName="maas-api" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967628 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967634 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd6e878-8025-4f8f-9a4f-acad58402755" containerName="cleanup" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967639 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="687df1f5-270f-4afe-afb5-472c0addf82f" containerName="istio-proxy" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967645 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29d5aa4a-29f9-45ca-a851-2886cff4cd40" containerName="cleanup" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967651 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d14ac054-aedd-4268-acb7-c83307b87b3b" containerName="cleanup" Jul 07 19:50:19.967680 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.967657 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3ae93d-8423-4c02-857a-fdadcaf41196" containerName="maas-api" Jul 07 19:50:19.972222 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.972206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:19.976563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.976542 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jpkhx\"/\"default-dockercfg-x9f5v\"" Jul 07 19:50:19.976666 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.976616 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jpkhx\"/\"openshift-service-ca.crt\"" Jul 07 19:50:19.977345 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.977330 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jpkhx\"/\"kube-root-ca.crt\"" Jul 07 19:50:19.994169 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:19.994144 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r"] Jul 07 19:50:20.140268 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.140237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-lib-modules\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.140429 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.140281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-proc\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.140429 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.140304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csn6t\" (UniqueName: \"kubernetes.io/projected/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-kube-api-access-csn6t\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.140429 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.140357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-podres\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.140429 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.140418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-sys\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.240974 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.240895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-podres\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.240974 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.240957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-sys\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.241502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.241007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-lib-modules\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.241502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.241041 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-podres\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.241502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.241056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-proc\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.241502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.241074 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-sys\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.241502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.241088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csn6t\" (UniqueName: \"kubernetes.io/projected/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-kube-api-access-csn6t\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.241502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.241115 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-lib-modules\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.241502 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.241149 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-proc\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.250624 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.250606 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csn6t\" (UniqueName: \"kubernetes.io/projected/dd906a49-dd81-4ea9-89fc-b1168cfe17e8-kube-api-access-csn6t\") pod \"perf-node-gather-daemonset-n5z8r\" (UID: \"dd906a49-dd81-4ea9-89fc-b1168cfe17e8\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.283053 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.283030 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.418172 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.418149 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r"] Jul 07 19:50:20.420249 ip-10-0-132-59 kubenswrapper[2577]: W0707 19:50:20.420222 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddd906a49_dd81_4ea9_89fc_b1168cfe17e8.slice/crio-0f9d38da99b6ca4a94a63666da18a3e469645b8e22cb9b9fd235fbc926857cf4 WatchSource:0}: Error finding container 0f9d38da99b6ca4a94a63666da18a3e469645b8e22cb9b9fd235fbc926857cf4: Status 404 returned error can't find the container with id 0f9d38da99b6ca4a94a63666da18a3e469645b8e22cb9b9fd235fbc926857cf4 Jul 07 19:50:20.786601 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.786566 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" event={"ID":"dd906a49-dd81-4ea9-89fc-b1168cfe17e8","Type":"ContainerStarted","Data":"1804409e3d27eae672b62ab51aa16118bccadfc7cdd299b1cfce5326e12007b1"} Jul 07 19:50:20.786764 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.786625 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" event={"ID":"dd906a49-dd81-4ea9-89fc-b1168cfe17e8","Type":"ContainerStarted","Data":"0f9d38da99b6ca4a94a63666da18a3e469645b8e22cb9b9fd235fbc926857cf4"} Jul 07 19:50:20.786764 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.786647 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:20.815593 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:20.815547 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" podStartSLOduration=1.815534564 podStartE2EDuration="1.815534564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:50:20.81422225 +0000 UTC m=+2526.749847609" watchObservedRunningTime="2026-07-07 19:50:20.815534564 +0000 UTC m=+2526.751159924" Jul 07 19:50:24.194695 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:24.194645 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w94kr_819038a7-6012-4c2f-964a-6f2c0e8b3359/dns/0.log" Jul 07 19:50:24.229238 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:24.229211 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w94kr_819038a7-6012-4c2f-964a-6f2c0e8b3359/kube-rbac-proxy/0.log" Jul 07 19:50:24.379510 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:24.379484 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pw2dn_2c902040-dd95-4b77-912d-2f2264683405/dns-node-resolver/0.log" Jul 07 19:50:25.146206 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:25.146177 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-snngx_a4ec0818-7ad9-4cd2-8511-dd5aa2fe666a/node-ca/0.log" Jul 07 19:50:26.236809 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:26.236774 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4wh7sr_f44ed616-49f7-49ef-8db4-ac4135c3379b/istio-proxy/0.log" Jul 07 19:50:26.542046 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:26.542016 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6nj7_93df3e10-d156-40e0-8f81-367099052b5f/istio-proxy/0.log" Jul 07 19:50:26.603268 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:26.603236 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6nj7_93df3e10-d156-40e0-8f81-367099052b5f/istio-proxy/1.log" Jul 07 19:50:26.672124 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:26.672096 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-chkp7_3c2fb729-32ba-4b76-acb7-ab6c74fe1074/payload-pre-processing/0.log" Jul 07 19:50:26.801537 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:26.801468 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-n5z8r" Jul 07 19:50:27.592165 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:27.592140 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rl9h8_1559affd-634a-494c-935c-8e553c347f86/serve-healthcheck-canary/0.log" Jul 07 19:50:28.293183 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:28.293151 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kgnws_deda2e77-de33-44a9-840e-bdc31a761927/kube-rbac-proxy/0.log" Jul 07 19:50:28.332395 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:28.332369 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kgnws_deda2e77-de33-44a9-840e-bdc31a761927/exporter/0.log" Jul 07 19:50:28.379012 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:28.378983 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kgnws_deda2e77-de33-44a9-840e-bdc31a761927/extractor/0.log" Jul 07 19:50:30.309246 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:30.309218 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4wh7sr_f44ed616-49f7-49ef-8db4-ac4135c3379b/istio-proxy/0.log" Jul 07 19:50:30.566306 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:30.566216 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6nj7_93df3e10-d156-40e0-8f81-367099052b5f/istio-proxy/0.log" Jul 07 19:50:30.626284 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:30.626257 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6nj7_93df3e10-d156-40e0-8f81-367099052b5f/istio-proxy/1.log" Jul 07 19:50:30.674437 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:30.674405 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-chkp7_3c2fb729-32ba-4b76-acb7-ab6c74fe1074/payload-pre-processing/0.log" Jul 07 19:50:31.074063 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:31.074037 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-67ccf_468aba6c-4436-470a-a5de-34c2e24325b0/manager/0.log" Jul 07 19:50:31.327769 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:31.327685 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tt9z9_d921ac91-a173-4000-9da8-26c6037abeaa/manager/0.log" Jul 07 19:50:31.417934 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:31.417899 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6d8c95d7dc-4wkl6_f3e3e2ee-1ea8-4687-a389-bcc6e10515b4/maas-api/0.log" Jul 07 19:50:31.517176 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:31.517150 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724225-2j7d5_d14ac054-aedd-4268-acb7-c83307b87b3b/cleanup/0.log" Jul 07 19:50:31.683000 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:31.682912 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9c77bd8b4-lpqf7_acbc8b6a-958b-483e-b393-1d6b17ed660f/manager/0.log" Jul 07 19:50:31.894563 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:31.894538 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8689849645-cjrtk_20d98418-bc55-46ad-a0a2-934ac44de34d/manager/0.log" Jul 07 19:50:31.930234 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:31.930206 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8689849645-cjrtk_20d98418-bc55-46ad-a0a2-934ac44de34d/copy-manifests/0.log" Jul 07 19:50:32.070678 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:32.070642 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4wh7sr_f44ed616-49f7-49ef-8db4-ac4135c3379b/istio-proxy/0.log" Jul 07 19:50:32.263490 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:32.263460 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-2bhbv_3adf1e3e-26fb-465c-b13c-cf417fc4461e/postgres/0.log" Jul 07 19:50:32.357977 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:32.357859 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6nj7_93df3e10-d156-40e0-8f81-367099052b5f/istio-proxy/0.log" Jul 07 19:50:32.410408 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:32.410369 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-d6nj7_93df3e10-d156-40e0-8f81-367099052b5f/istio-proxy/1.log" Jul 07 19:50:32.458217 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:32.458191 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-chkp7_3c2fb729-32ba-4b76-acb7-ab6c74fe1074/payload-pre-processing/0.log" Jul 07 19:50:33.683095 ip-10-0-132-59 kubenswrapper[2577]: I0707 19:50:33.683050 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d95ccd586-snz76_54ab0ba6-dbe8-4923-8bdb-974b4c1d62d1/manager/0.log"